Unix
Contents
- 1 Iptables
- 2 smtp - sending email manually
- 3 General UNIX How-to's
Iptables
smtp - sending email manually
General UNIX How-to's
Sudo grep *
normal grep wont work if using a wildcard, generally returns `No such file or directory`
sudo sh -c "grep 'SEARCHSTRING' /var/lib/blah/*"
Related -- Return lines before or after a match
grep -A1 '1_after_this_match' grep -B1 '1_line_before_match'
Octal File permissions
stat -c '%A %a %n' * -rw-r--r-- 644 /etc/passwd
Read = 4 Write = 2 Execute = 1
Recursive write to directories but do not set files to be executable
chmod -R a=rwX
Sticky bit (only owner can delete directory)
chmod 0+t chmod 1xyz (leading 1 = sticky bit)
Network command changes
nslookup --> getent hosts google.com
bash
Find when the latest sub-directory was created given a path and search pattern (aka datachannel done files..)
date; bash -c 'for DIRECTORY in `find /appl/proviso/data/datachannel/ -name done -type d -user pvuser`; do echo "==";echo "$DIRECTORY"; ls -rt $DIRECTORY | tail -1 ; done'
Count the files waiting to be processed
date; find /appl/proviso/data/datachannel/BLB*/output | cut -d "/" -f 6 | sort | uniq -c
Log to syslog / /var/log/messages
echo "hello world" logger -t firstscript "hello world"
Run shell command every x seconds
bash -c 'while [[ 0 ]]; do echo "----"; date '+%H:%M:%S'; ls -l /appl/proviso/datachannel/BLB.2.251/output; echo "----"; sleep 5; done'
grep with OR
egrep -e 'JMS''PROCESS|SAM''CONNECT'
For the locker below - better to use extended ps output -incase the filename gets truncated (works on spac)
/usr/ucb/ps -xwww
how to get the full path to the executable
% ps -ef | grep [[processname]] root 24466 0.0 0.0 1476 280 ? S 2009 0:00 supervise sshd % sudo ls -l /proc/24466/exe
Bash script to force locking / only one instance of a script will run at a time
#!/bin/bash # # watchdog # # Run as a cron job to keep an eye on what_to_monitor which should always # be running. Restart what_to_monitor and send notification as needed. # NAME=dc_log_sam_blb_errors.pl ENV=/appl/proviso/datachannel/dataChannel.env APATH=/appl/proviso/Application/ExtAdapter/SAM_MONITOR PARAM=/appl/proviso/datachannel/log/proviso.log GREP=/bin/grep PS=/bin/ps $PS -ef|$GREP -v grep|$GREP $NAME >/dev/null 2>&1 case "$?" in 0) # It is running in this case so we do nothing. ;; 1) echo "$NAME is NOT RUNNING. Starting $NAME" . $ENV cd $APATH ./$NAME $PARAM 2>&1 >/dev/null & ;; esac exit
Remove files EDS put in temp and never remove (cronjob as root)
## remove old EDS files from malformed scripts that run 00 05 * * * find /tmp/CTSA* -user root -ctime +2 -type f -exec rm -f {} \; 2>&1 00 05 * * * find /tmp/disktacct* -user root -ctime +2 -type f -exec rm -f {} \; 2>&1 00 05 * * * find /tmp/locked* -user root -ctime +2 -type f -exec rm -f {} \; 2>&1
Keep a process running after you log off - with the right flags and run as root a process can never be stopped! Cli output will be written to nohup.out in the directory you run the command from
nohup <command> &
Bash calling python
#!/bin/bash echo Welcome to our shell script python <<__EOF_PYTHON_SCRIPT print ("Howdy from Python!") __EOF_PYTHON_SCRIPT echo "And we are back!"
Defunct Processes - Find and map their parents
/usr/proc/bin/ptree `ps -ef | grep -v ^\ \ \ \ root | awk '/<defunct>/ {print $2}'` | less
Symbolic Link
ln -s {/path/to/file-name} {link-name} rm {link-name}
Send a text file via email from unix
cat <file> | mailx -s 'sunject' <email@dot.com>
Start Apache (if some retard didnt create the init.d file)
/usr/local/apache2/bin/httpd -k start
Using zip instead of tar (for files with REALLY long names) will output myzipfile.zip containing all data in directory (within directory folder in the zip)
zip -r myzipfile directory
Find the total size of all files in this folder like abc
du -ks /appl/proviso/dataload/log/*.pcap | awk '{t+=$1}END{printf "%.2f Mb\n", t/1024}' Remember to escape the $ " and \ ./run_command_on_dataloaders.sh " du -ks /appl/proviso/dataload/log/*.pcap | awk '{t+=\$1}END{printf \"%.2f Mb\", t/1024}' && ls -l /appl/proviso/dataload/log/*.pcap | wc -l"
Log Memory Changes for a Process run with NCO_MEM.sh PID
#! /bin/sh ############################################### # Script to log memory usage over time ############################################### export PLATFORM PLATFORM=`uname -p` ############################################### # Edit SLEEP as required ############################################### SLEEP=10 # if [ $# -eq 1 ] then PID=$1 else echo "Usage : `basename $0` [PID}" echo "SLEEP=$SLEEP" exit fi export DATE PS PID SLEEP OLD_MEM MEM OLD_VMEM VMEM OLD_RSS RSS echo "PLATFORM=$PLATFORM" ############################################### ############################################### if [ "$PLATFORM" = "" ] then echo "Unsupported platform" exit ############################################### # LINUX ############################################### elif [ "$PLATFORM" = "x86_64" ] then PS=`ps -p $PID -o "size rss vsz pmem time pcpu" | head -1` date '+%d/%m/%y %H:%M:%S' echo "Date : $PS" OLD_MEM=0 OLD_VMEM=0 OLD_RSS=0 while true do MEM=`ps -p $PID -o size | tail -1` VMEM=`ps -p $PID -o vsz | tail -1` RSS=`ps -p $PID -o rss | tail -1` if [ $MEM -ne $OLD_MEM -o $VMEM -ne $OLD_VMEM -o $RSS -ne $OLD_RSS ] then OLD_VMEM=$VMEM OLD_MEM=$MEM OLD_RSS=$RSS DATE=`date '+%y%m%d%H%M%S'` PS=`ps -p $PID -o "size rss vsz pmem time pcpu" | tail -1` echo "$DATE : $PS" fi sleep $SLEEP done ############################################### # Solaris ############################################### elif [ "$PLATFORM" = "sparc" ] then PS=`ps -p $PID -o "osz rss vsz pmem time pcpu" | head -1` date '+%d/%m/%y %H:%M:%S' echo "Date : $PS" OLD_MEM=0 OLD_VMEM=0 OLD_RSS=0 while true do MEM=`ps -p $PID -o osz | tail -1` VMEM=`ps -p $PID -o vsz | tail -1` RSS=`ps -p $PID -o rss | tail -1` if [ $MEM -ne $OLD_MEM -o $VMEM -ne $OLD_VMEM -o $RSS -ne $OLD_RSS ] then OLD_VMEM=$VMEM OLD_MEM=$MEM OLD_RSS=$RSS DATE=`date '+%y%m%d%H%M%S'` DATE=`date '+%d/%m/%y %H:%M:%S'` PS=`ps -p $PID -o "osz rss vsz pmem time pcpu" | tail -1` echo "$DATE : $PS" fi sleep $SLEEP done ############################################### # AIX ############################################### elif [ "$PLATFORM" = "powerpc" ] then #PS=`ps -p $PID -o "osz rss vsz pmem time pcpu" | head - PS=`ps -p $PID -o "rssize vsz pmem" | head -1` date '+%d/%m/%y %H:%M:%S' echo "Date : $PS" OLD_MEM=0 OLD_VMEM=0 while true do MEM=`ps -p $PID -o rssize | tail -1` VMEM=`ps -p $PID -o vsz | tail -1` if [ $MEM -ne $OLD_MEM -o $VMEM -ne $OLD_VMEM ] then OLD_VMEM=$VMEM OLD_MEM=$MEM DATE=`date '+%y%m%d%H%M%S'` DATE=`date '+%d/%m/%y %H:%M:%S'` PS=`ps -p $PID -o "rssize vsz pmem" | tail -1` echo "$DATE : $PS" fi sleep $SLEEP done ############################################### # Last FI ############################################### fi #EOF
Format the xml files in the current directory
bash -c 'for XML in `find *.xml`; do echo "$XML"; xmllint -format $XML > $XML.formatted; done'
SED and REGEX on the command line
cat blah | cut -d " " -f4 | sed 's/[0-9]*//g'
www.tutorialspoint.com/unix/unix-regular-expressions.htm
CUT and CUT -ordered
Normal cut will seperate on delimiter -d ";" and print fields -f 1,2-10 quickly but will only do them in order.
To be able to specify the output in a specific order use awk
awk -F";" '{print $9,$24,$34,$4,$11,$12,$16,$10,$13,$14,$15,$23,$33,$5,$6,$7,$19,$20,$8,$22,$36,$26,$35,$25,$1,$2,$17,$18,$27,$28,$29,$30,$32,$31,$21,$37,$38,$3}' properties_new.csv
remove a file with a strange name
$ ls -rlt total 2018 drwxrwxrwx 2 pvuser proviso 512 Jan 23 14:56 old -rw-r----- 1 pvuser proviso 1018744 Jan 23 16:03 OSS_Inventory.csv -rw-rw-rw- 1 pvuser proviso 0 Jan 23 17:04 --? -rw-rw-rw- 1 pvuser proviso 0 Jan 23 17:04 -help $ rm -- --? $ ls -l total 2018 -rw-rw-rw- 1 pvuser proviso 0 Jan 23 17:04 -help -rw-r----- 1 pvuser proviso 1018744 Jan 23 16:03 OSS_Inventory.csv drwxrwxrwx 2 pvuser proviso 512 Jan 23 14:56 old
Find open files on a partition
lsof will list open file and sockets
lsof
If lsof is not installed - or you dont have rights..
to find active processes for a filesystem do "fuser -c filesystem" [sf2395:root]:/appl/proviso/log > fuser -c /appl/proviso/log /appl/proviso/log: 25190c 21255c 24996c 18286o 6915o 5473o 4816o [sf2395:root]:/appl/proviso/log > then do ptree of the pid above to see what they're doing: [sf2395:root]:/appl/proviso/log > ptree 18286 1440 /usr/sbin/cron 18151 sh -c . /appl/proviso/DataAccessAPI/mpoi/edr/run.sh > /dev/null 2>&1 18286 /appl/proviso/DataAccessAPI/j2sdk1.4.1_06/bin/java -classpath .:/appl/proviso/D [sf2395:root]:/appl/proviso/log >
Netcat - pipe a unix input to another machines socket - or listen to a socket and pipe the output.
www.tutorialspoint.com/unix_commands/nc.htm
www.g-loaded.eu/2006/11/06/netcat-a-couple-of-useful-examples/
nc - arbitrary TCP and UDP connections and listens
* nc -words [-46Ddhklnr StUuvz] [-i interval] [-p source'port] [-s source'ip'address] [-T To S] [-w timeout] [-X proxy'protocol] [Xo -x proxy_address[: port]] [hostname] [port[s]] TCP Server / Listener nc -n <port> TCP Client nc -n <ipaddress> <port> some OS you must specify port with -p (thanks centos)
DESCRIPTION
The nc (or netcat) utility is used for just about anything under the sun involving TCP or UDP. It can open TCP connections, send UDP packets, listen on arbitrary TCP and UDP ports, do port scanning, and deal with both IPv4 and IPv6. Unlike telnet(1), nc scripts nicely, and separates error messages onto standard error instead of sending them to standard output, as telnet(1) does with some.
- Common uses include
- simple TCP proxies
shell-script based HTTP clients and servers network daemon testing a SOCKS or HTTP Proxy Command for ssh(1) and much, much more
Crontab Syntax
Im always forgetting the day of week part
* * * * * command to be executed - - - - - | | | | | | | | | +----- day of week (0 - 6) (Sunday=0) | | | +------- month (1 - 12) | | +--------- day of month (1 - 31) | +----------- hour (0 - 23) +------------- min (0 - 59)
Rsync Remote Backup
This page Cygwin_Rsync details how to backup a Windows XP machine to another Windows XP machine over the internet. (using SSH + Rsync + Cygwin)
Wireshark / tcpflow
Tcpflow is a good command for putting tcp sessions back together, the following command will do this and remove the report.xml
tcpflow -T%t-%A-%a -x netviz -i [interface] -S enable_report=NO -o [output_dir] [wireshark collection filter]
TCP Replay
Will resend udp and tcp packets
N.B tcpreplay must send packets out an interface (cant send to the host its running on, so two VM's required)
sudo apt install tcpreplay
tcprewrite --infile=original.cap --outfile=changed.cap --srcipmap=0.0.0.0/0:<MY HOST IP>/32 --dstipmap=0.0.0.0/0:<MY HOST IP>/32 --enet-smac=<enp0s25 mac addr> --enet-dmac=<enp0s25 mac addr> --fixcsum sudo tcpreplay --intf1=ens33 changed.cap
Control Characters
grep '[[:cntrl:]]' /tmp/file.txt
X Windows
A helpful guide www.redwireservices.com/remote-x11-for-linux-unix
RHEL 6
Unlock an account
pam_tally2 --user=username --reset
Sticky Bits
https://www.liquidweb.com/kb/how-do-i-set-up-setuid-setgid-and-sticky-bits-on-linux/
Dig
- get the address(es) for yahoo.com
dig yahoo.com A +noall +answer
- get a list of yahoo's mail servers
dig yahoo.com MX +noall +answer
- get a list of DNS servers authoritative for yahoo.com
dig yahoo.com NS +noall +answer
- get all of the above
❯ dig wiretap.nz ANY +noall +answer ; <<>> DiG 9.10.6 <<>> wiretap.nz ANY +noall +answer ;; global options: +cmd wiretap.nz. 24 IN A 172.67.133.52 wiretap.nz. 24 IN A 104.21.13.215 wiretap.nz. 112 IN AAAA 2606:4700:3030::6815:dd7 wiretap.nz. 112 IN AAAA 2606:4700:3031::ac43:8534
- More obscurely, for the present anyway, you can also poll for a host’s IPv6 address using the AAAA option.
dig www.isc.org AAAA +short
- If the domain you want to query allows DNS transfers, you can get those, too. The reality of life on the Internet, however, is that very few domains allow unrestricted transfers these days.
dig yourdomain.com AXFR
Hex to Decimal
printf "%d\n", 0xFF
Decimal to Hex
printf "%02X\n", 255