Remove line breaks in CSV

Problem:
You have a CSV lines with new line breaks and you need one line per entry.

Solution:

A simple solution could be:
sed ':a;N;$!ba;s/\r\n/ /g' myfile.csv > myfile_no_nline.csv
If the new lines inside the CSV body are Windows new lines (\r\n) and the CSV lines are UNIX (\n).

But normally you are not so lucky. So if your CSV new lines are between quotes (“) you can use the following command:
cat myfile.csv | gawk -v RS='"' 'NR % 2 == 0 { gsub(/\n/, "") } { printf("%s%s", $0, RT) }' > myfile_no_nline.csv

Advertisements

Continue after a broken SCP connection

Problem:

You transfer a file to a remote computer via scp:

scp my_favourite_file.tar.gz user@192.168.16.2:~

But before the transfer is completed the connection is broken.
How can I continue the transfer from the point it broke?

Solution:


rsync -P -e ssh my_favourite_file.tar.gz user@192.168.16.2:~

Minimum HTTP server with netcat

Problem

Sometimes you need to have a simple HTTP server. You want to check connectivity from other machine or whatever sinister reason you can have.

Solution

We are going to create a simple HTTP server at port 8000.
You can use the netcat command:

nc -l -p 8000 -c 'echo -e "HTTP/1.1 200 OK\n\nHello $(id -un)\nCurrent time $(date)"

Then you can go to a web browser and write load the url at the port 8000

http://localhost:8000

But this works just for one connection, after that connection you must execute again the nc command, so to accept more connections you must add a while loop:

while [ 1 ];do nc -l -p 8000 -c 'echo -e "HTTP/1.1 200 OK\n\nHello $(id -un)\nCurrent time $(date)"'; done

May be you want to display a simple html file (with no images, ajax or any other HTML element that launchs more HTTP requests) like this one:

<!DOCTYPE html>
<html>
<body>
<h1>Example</h1>
This is a simple<br/>
webserver running with
<b>nc</b>
</body>
</html>

Let’s suppose that you have your html saved in a file called index.html, then you can show it in your netcat server as follows:

while [ 1 ];do nc -l -p 8000 -c 'echo -e "HTTP/1.1 200 OK\n";cat index.html'; done

If you are not able to connect to your netcat server may be you have activated iptables, take a look writing as root:

iptables -L

If you want to remove the iptables that can be blocking this requests write the following commands as root:

iptables -F
iptables -X
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT