KnowUbuntu

Just a quick post here. I was messing around with some pretty large text files yesterday. I had a file with 2.9 million insert statements (one per line) that I was running on a database. It was taking forever so I stopped it. I new how many of the insert statements had run so I needed to get rid of the top million and a half so that they were not inserted again. I came across this nifty one-liner that made it really easy and didn’t take too long: sed '1,1500000d' inserts.sql > inserts2.sql Now I will explain what I understand it to be doing. sed is a command that is very powerful (I don’t know the first thing about using it really) for working with text and files. The 1,1500000 indicates that I want to do something to lines 1-1500000 of the file. The d tells sed that I want to delete those lines. Then specify the file you want to do that do and output the remaining contents (>) to a new file insert2.sql.

Pretty simple, but sed can be pretty involved.

Here is a bash script one-liner for printing out a random line from a file.

j=$(wc -l words.txt | awk '{print $1}'); i=$RANDOM; i=$(echo "$i%$j+1" | bc); sed -n ${i}p words.txt

The variable j has stored in it the total number of lines in the file with the help of the wc and awk commands. The variable i has stored in it a random number that is then changed to be in between 1 and the number of lines in the file, j. The random number is obtained by assigning i to $RANDOM and then piping a mod operation to the bc calculator. The resulting random line number, i, is displayed using the sed command.

PS: I know the script is on two lines, that that is just because there wasn’t room on the blog to go all the way across.