Some times you create a file like password.properties, that you commit with fake values. But then you want to update the content of that file and be sure that is never pushed to the remote.
To do that you have to follow the next steps:
Given several partitioned AVRO formatted files, together with the AVSC schema, we want to create a table in Hive.
We have hundreds of files in a directory partitioned by year and mont in the HDFS folder /data/mytable.db/mytable.
The folder structure is:
/data/mytable.db/mytable/Year=2018/month=11
/data/mytable.db/mytable/Year=2018/month=12
/data/mytable.db/mytable/Year=2019/month=1
/data/mytable.db/mytable/Year=2019/month=2
/data/mytable.db/mytable/Year=2019/month=3
/data/mytable.db/mytable/Year=2019/month=4
How to configure docker behind an enterprise firewall
We are going to learn how to create an R package with unit testing and documentation.
We will use the packages devtools, usethis and roxygen2.
It is possible to create the jobs in jenkins using Groovy.
You can create a Groovy script file $JENKINS_HOME/init.groovy, or any .groovy file in the directory $JENKINS_HOME/init.groovy.d/, to run some additional things right after Jenkins starts up.
In Ubuntu $JENKINS_HOME is by default configured in /var/lib/jenkins/
Sometimes you want to move docker images from one machine to another and you don’t have a proper docker registry or it is not availabe.
Docker provides commands for that:
But these images can be really big in size, there is an alternative to save only the current container, no the image with all its historic data.
Sometimes the default Docker IP has to be changed.
You can use openconnect to connect to a Juniper VPN:
openconnect --juniper mivpn.es
It will parse the web and will ask you for user/password.
But sometimes this does not work, so you would have to login via web and then copy the cookie DSID:
openconnect --juniper -C "DSID=ab353bbbcd677e8adf5a2f335b6c77ac" mivpn.es
Problem:
You have a CSV lines with new line breaks and you need one line per entry.
Solution:
A simple solution could be:
sed ':a;N;$!ba;s/\r\n/ /g' myfile.csv > myfile_no_nline.csv
If the new lines inside the CSV body are Windows new lines (\r\n) and the CSV lines are UNIX (\n).
But normally you are not so lucky. So if your CSV new lines are between quotes (“) you can use the following command:
cat myfile.csv | gawk -v RS='"' 'NR % 2 == 0 { gsub(/\n/, "") } { printf("%s%s", $0, RT) }' > myfile_no_nline.csv
Problem:
You transfer a file to a remote computer via scp:
scp my_favourite_file.tar.gz user@192.168.16.2:~
But before the transfer is completed the connection is broken.
How can I continue the transfer from the point it broke?
Solution:
rsync -P -e ssh my_favourite_file.tar.gz user@192.168.16.2:~
Small tips that I don't want to forget 