All Questions
Tagged with text-processing csv
373 questions
8
votes
5
answers
1k
views
Run command on each line of CSV file, using fields in different places of the command
I have a CSV file and want to run a command for each line, using the fields of the file as separate arguments.
For example given the following file:
foo,42,red
bar,13,blue
baz,27,green
I want to run ...
- 272
-4
votes
5
answers
127
views
Command to display all the employees whose first name have more than 6 characters
From the script below I need to know the following:
EmpNo#Email#Name#JobLevel#Experience
641357#Amrit_Mohanty#Amrit Mohanty#3#2
678522#Puneet_Mishra#Puneet Mishra#3#1
670242#Vikas_Bharti#Vikas Bharti#...
- 183
3
votes
3
answers
355
views
Large file manipulation
I have a Garmin Nuvi which uses OpenStreet maps. Garmin do postcodes, but is usually 2-3 years out of date for Scotland. OSM does not do British postcodes, but the Post office does, and it can be ...
- 49
-3
votes
2
answers
100
views
Using awk, append a value in last field in csv file based upon value in specific field
input.csv -
"family_guy","Brian","b47f0a80-d848-4d81-a45b-7ba930e6048b","son"
"family_guy","Brian","b47f0a80-d848-4d81-a45b-...
- 553
1
vote
1
answer
105
views
Print csv columns according to user input
I have a .csv file with multiple lines with the following format:
¬Country¬,¬Year¬,¬Singer¬,¬Song Title¬
¬Japan¬,¬1999¬,¬Utada Hikaru¬,¬First Love¬
¬South Korea¬,¬1999¬,¬Lee Jung Hyun¬,¬Wa¬
...
I can ...
- 27
3
votes
5
answers
193
views
How to remove double quotes within the double-quoted field values in .dat file
I have a text file that has around 15 columns.
The fields are separated by comma.
One column that is description is double-quoted
and also has some words which are double-quoted.
I need to retain ...
- 43
2
votes
7
answers
2k
views
How to remove embedded newlines from CSV fields
I am struggling with processing a file. Normally I should have a line for each host definition. But sometimes, someone split some fields on different lines.
Here is an example:
"host1","...
5
votes
5
answers
818
views
Remove lines from a csv file based on column value
I have a csv file with 12 million lines in the following format:
mcu_i,INIT,200,iFlash, 11593925, 88347,,0x00092684,r,0x4606b570, ok,, 32,single,op-c,0,, 0, 0, 0,...
- 84
-1
votes
4
answers
317
views
awk command to extract columns from a csv file whose header contains a certain substring
I have a huge csv file that is of the format
aaa1, "aaa2, aa214", aa21, "aa, a14", aa211, aa44, aaa445
data, data, data, data, data, data, data,
......................................
- 254
3
votes
5
answers
478
views
How to create a new column and add a random identifier to it with miller
I want to add a column with a randomly created "case number" to my csv file. The first 2 letters of the casenumber must be any letter from A-Z in capitals. followed by 5 random numbers.
...
- 357
-1
votes
1
answer
195
views
Create new CSV files based on latest timestamp data
I have a script that accesses multiple dat files and produce csv files based on previous day’s data. these DAT files are updated on per minute bases with data from various instruments.
Script snippet:
...
- 151
0
votes
3
answers
331
views
Convert lower-case to uppercase with the output to a new column using miller
I want to copy column NAME to column NAME-LOWERCASE. NAME-LOWERCASE should only contain lowercase letters. The uppercase should be untouched in all column expect NAME-LOWERCASE
input
NAME,test
PTC,N
...
- 357
0
votes
1
answer
128
views
Need Help with Date expression - AWK in a csv file
Situation: csv file has date column with different date formats, I want to convert them into a unambiguous date format (like +"%m-%d-%Y").
Example File data : File name = Date_Test_new.csv
3/...
3
votes
5
answers
1k
views
Replace a block of spaces with a comma
I have a file in txt format that I want to convert to csv. Between each field there are blocks of spaces. The amount of spaces is not the same between every field.
Is there a command with sed or awk ...
- 357
0
votes
1
answer
370
views
Process CSV file through command line: Remove only middle rows between consecutive row entries if consecutive entries have the same 2nd column value
I have a CSV file with only 2 columns (but lots of rows) and the occasional irregular row which always starts with an asterisk (*) character and may span more than two columns. Using just the Linux ...
- 161