Unix

From neil.tappsville.com
Jump to navigationJump to search


Iptables

iptables

smtp - sending email manually

smtp email

General UNIX How-to's

Sudo grep *

normal grep wont work if using a wildcard, generally returns `No such file or directory`

sudo sh -c "grep 'SEARCHSTRING' /var/lib/blah/*"

Related -- Return lines before or after a match

grep -A1 '1_after_this_match'
grep -B1 '1_line_before_match'

Octal File permissions

stat -c '%A %a %n' *
-rw-r--r-- 644 /etc/passwd
Read = 4
Write = 2 
Execute = 1

Recursive write to directories but do not set files to be executable

chmod -R a=rwX

Sticky bit (only owner can delete directory)

chmod 0+t
chmod 1xyz (leading 1 = sticky bit)

Run as owner (only settable to files)

u+s
chmod 4xyz

Run as group

g+s
chmod 2xyz

Network command changes

nslookup --> getent hosts google.com

bash

Find when the latest sub-directory was created given a path and search pattern (aka datachannel done files..)

date; bash -c 'for DIRECTORY in `find /appl/proviso/data/datachannel/ -name done -type d -user pvuser`; do echo "==";echo "$DIRECTORY"; ls -rt $DIRECTORY | tail -1 ;  done'

Count the files waiting to be processed

date; find /appl/proviso/data/datachannel/BLB*/output  | cut -d "/" -f 6 | sort | uniq -c

Log to syslog / /var/log/messages

echo "hello world"
logger -t firstscript "hello world"

Run shell command every x seconds

bash -c 'while [[ 0 ]]; do echo "----"; date '+%H:%M:%S'; ls -l /appl/proviso/datachannel/BLB.2.251/output; echo "----"; sleep 5; done'

grep with OR

egrep -e 'JMS''PROCESS|SAM''CONNECT'

For the locker below - better to use extended ps output -incase the filename gets truncated (works on spac)

/usr/ucb/ps -xwww

how to get the full path to the executable


% ps -ef | grep [[processname]]
root     24466  0.0  0.0   1476   280 ?        S     2009   0:00 supervise sshd
% sudo ls -l /proc/24466/exe


Bash script to force locking / only one instance of a script will run at a time

#!/bin/bash
#
# watchdog
#
# Run as a cron job to keep an eye on what_to_monitor which should always
# be running. Restart what_to_monitor and send notification as needed.
#

NAME=dc_log_sam_blb_errors.pl
ENV=/appl/proviso/datachannel/dataChannel.env
APATH=/appl/proviso/Application/ExtAdapter/SAM_MONITOR
PARAM=/appl/proviso/datachannel/log/proviso.log
GREP=/bin/grep
PS=/bin/ps

$PS -ef|$GREP -v grep|$GREP $NAME >/dev/null 2>&1
case "$?" in
 0)
 # It is running in this case so we do nothing.
 ;;
 1)
 echo "$NAME is NOT RUNNING. Starting $NAME"
 . $ENV
 cd $APATH
 ./$NAME $PARAM 2>&1 >/dev/null &
 ;;
esac

exit

Remove files EDS put in temp and never remove (cronjob as root)

## remove old EDS files from malformed scripts that run
00 05 * * * find /tmp/CTSA* -user root -ctime +2 -type f -exec rm -f {} \; 2>&1
00 05 * * * find /tmp/disktacct* -user root -ctime +2 -type f -exec rm -f {} \; 2>&1
00 05 * * * find /tmp/locked* -user root -ctime +2 -type f -exec rm -f {} \; 2>&1

Keep a process running after you log off - with the right flags and run as root a process can never be stopped! Cli output will be written to nohup.out in the directory you run the command from

nohup <command> &

Bash calling python

#!/bin/bash
echo Welcome to our shell script
 
python <<__EOF_PYTHON_SCRIPT
print ("Howdy from Python!")
__EOF_PYTHON_SCRIPT
 
echo "And we are back!"

Defunct Processes - Find and map their parents

/usr/proc/bin/ptree `ps -ef | grep -v ^\ \ \ \ root | awk '/<defunct>/ {print $2}'` | less

Symbolic Link

ln -s {/path/to/file-name} {link-name}
rm {link-name}


Send a text file via email from unix

cat <file> | mailx -s 'sunject' <email@dot.com>

Start Apache (if some retard didnt create the init.d file)

/usr/local/apache2/bin/httpd -k start

Using zip instead of tar (for files with REALLY long names) will output myzipfile.zip containing all data in directory (within directory folder in the zip)

zip -r myzipfile directory


Find the total size of all files in this folder like abc

du -ks /appl/proviso/dataload/log/*.pcap | awk '{t+=$1}END{printf "%.2f Mb\n", t/1024}'

Remember to escape the $ " and  \

 ./run_command_on_dataloaders.sh " du -ks /appl/proviso/dataload/log/*.pcap | awk '{t+=\$1}END{printf \"%.2f Mb\", t/1024}' && ls -l /appl/proviso/dataload/log/*.pcap | wc -l"

Log Memory Changes for a Process run with NCO_MEM.sh PID

#! /bin/sh
###############################################
# Script to log memory usage over time
###############################################
export PLATFORM
PLATFORM=`uname -p`
###############################################
# Edit SLEEP as required
###############################################
SLEEP=10
#
if [ $# -eq 1 ]
then
PID=$1
else
echo "Usage : `basename $0` [PID}"
echo "SLEEP=$SLEEP"
exit
fi
export DATE PS PID SLEEP OLD_MEM MEM OLD_VMEM VMEM OLD_RSS RSS
echo "PLATFORM=$PLATFORM"
###############################################
###############################################
if [ "$PLATFORM" = "" ]
then
echo "Unsupported platform"
exit
###############################################
# LINUX
###############################################
elif [ "$PLATFORM" = "x86_64" ]
then
PS=`ps -p $PID -o "size rss vsz pmem time pcpu" | head -1`
date '+%d/%m/%y %H:%M:%S'
echo "Date         : $PS"
OLD_MEM=0
OLD_VMEM=0
OLD_RSS=0
while true
do
MEM=`ps -p $PID -o size | tail -1`
VMEM=`ps -p $PID -o vsz | tail -1`
RSS=`ps -p $PID -o rss | tail -1`
if [ $MEM -ne $OLD_MEM -o $VMEM -ne $OLD_VMEM -o $RSS -ne $OLD_RSS ]
then
OLD_VMEM=$VMEM
OLD_MEM=$MEM
OLD_RSS=$RSS
DATE=`date '+%y%m%d%H%M%S'`
PS=`ps -p $PID -o "size rss vsz pmem time pcpu" | tail -1`
echo "$DATE : $PS"
fi
sleep $SLEEP
done
###############################################
# Solaris
###############################################
elif [ "$PLATFORM" = "sparc" ]
then
PS=`ps -p $PID -o "osz rss vsz pmem time pcpu" | head -1`
date '+%d/%m/%y %H:%M:%S'
echo "Date         : $PS"
OLD_MEM=0
OLD_VMEM=0
OLD_RSS=0
while true
do
MEM=`ps -p $PID -o osz | tail -1`
VMEM=`ps -p $PID -o vsz | tail -1`
RSS=`ps -p $PID -o rss | tail -1`
if [ $MEM -ne $OLD_MEM -o $VMEM -ne $OLD_VMEM -o $RSS -ne $OLD_RSS ]
then
OLD_VMEM=$VMEM
OLD_MEM=$MEM
OLD_RSS=$RSS
DATE=`date '+%y%m%d%H%M%S'`
DATE=`date '+%d/%m/%y %H:%M:%S'`
PS=`ps -p $PID -o "osz rss vsz pmem time pcpu" | tail -1`
echo "$DATE : $PS"
fi
sleep $SLEEP
done
###############################################
# AIX
###############################################
elif [ "$PLATFORM" = "powerpc" ]
then
#PS=`ps -p $PID -o "osz rss vsz pmem time pcpu" | head -
PS=`ps -p $PID -o "rssize vsz pmem" | head -1`
date '+%d/%m/%y %H:%M:%S'
echo "Date         : $PS"
OLD_MEM=0
OLD_VMEM=0
while true
do
MEM=`ps -p $PID -o rssize | tail -1`
VMEM=`ps -p $PID -o vsz | tail -1`
if [ $MEM -ne $OLD_MEM -o $VMEM -ne $OLD_VMEM ]
then
OLD_VMEM=$VMEM
OLD_MEM=$MEM
DATE=`date '+%y%m%d%H%M%S'`
DATE=`date '+%d/%m/%y %H:%M:%S'`
PS=`ps -p $PID -o "rssize vsz pmem" | tail -1`
echo "$DATE : $PS"
fi
sleep $SLEEP
done
###############################################
# Last FI
###############################################
fi
#EOF


Format the xml files in the current directory

bash -c 'for XML in `find *.xml`; do echo "$XML"; xmllint -format $XML > $XML.formatted; done'


SED and REGEX on the command line

cat blah | cut -d " " -f4 | sed 's/[0-9]*//g'

www.tutorialspoint.com/unix/unix-regular-expressions.htm


CUT and CUT -ordered

Normal cut will seperate on delimiter -d ";" and print fields -f 1,2-10 quickly but will only do them in order.

To be able to specify the output in a specific order use awk

awk -F";" '{print $9,$24,$34,$4,$11,$12,$16,$10,$13,$14,$15,$23,$33,$5,$6,$7,$19,$20,$8,$22,$36,$26,$35,$25,$1,$2,$17,$18,$27,$28,$29,$30,$32,$31,$21,$37,$38,$3}' properties_new.csv

remove a file with a strange name

$ ls -rlt
total 2018
drwxrwxrwx   2 pvuser   proviso      512 Jan 23 14:56 old
-rw-r-----   1 pvuser   proviso  1018744 Jan 23 16:03 OSS_Inventory.csv
-rw-rw-rw-   1 pvuser   proviso        0 Jan 23 17:04 --?
-rw-rw-rw-   1 pvuser   proviso        0 Jan 23 17:04 -help
$ rm -- --?
$ ls -l
total 2018
-rw-rw-rw-   1 pvuser   proviso        0 Jan 23 17:04 -help
-rw-r-----   1 pvuser   proviso  1018744 Jan 23 16:03 OSS_Inventory.csv
drwxrwxrwx   2 pvuser   proviso      512 Jan 23 14:56 old

Find open files on a partition

lsof will list open file and sockets

lsof

If lsof is not installed - or you dont have rights..

to find active processes for a filesystem do "fuser -c filesystem"
[sf2395:root]:/appl/proviso/log > fuser -c /appl/proviso/log
/appl/proviso/log:    25190c   21255c   24996c   18286o    6915o    5473o    4816o
[sf2395:root]:/appl/proviso/log >


then do ptree of the pid above to see what they're doing:
[sf2395:root]:/appl/proviso/log > ptree 18286
1440  /usr/sbin/cron
  18151 sh -c . /appl/proviso/DataAccessAPI/mpoi/edr/run.sh > /dev/null 2>&1
    18286 /appl/proviso/DataAccessAPI/j2sdk1.4.1_06/bin/java -classpath .:/appl/proviso/D
[sf2395:root]:/appl/proviso/log >


Netcat - pipe a unix input to another machines socket - or listen to a socket and pipe the output.

www.tutorialspoint.com/unix_commands/nc.htm


www.g-loaded.eu/2006/11/06/netcat-a-couple-of-useful-examples/


nc - arbitrary TCP and UDP connections and listens


* nc  -words [-46Ddhklnr StUuvz] [-i interval] [-p source'port] [-s source'ip'address] [-T To S] [-w timeout] [-X proxy'protocol] [Xo -x proxy_address[: port]] [hostname] [port[s]]

TCP Server / Listener
nc -n <port>
TCP Client
nc -n <ipaddress> <port>

some OS you must specify port with -p (thanks centos)


DESCRIPTION The nc (or netcat) utility is used for just about anything under the sun involving TCP or UDP. It can open TCP connections, send UDP packets, listen on arbitrary TCP and UDP ports, do port scanning, and deal with both IPv4 and IPv6. Unlike telnet(1), nc scripts nicely, and separates error messages onto standard error instead of sending them to standard output, as telnet(1) does with some.

Common uses include 
simple TCP proxies

shell-script based HTTP clients and servers network daemon testing a SOCKS or HTTP Proxy Command for ssh(1) and much, much more


Crontab Syntax

Im always forgetting the day of week part

*     *     *     *     *  command to be executed
-     -     -     -     -
|     |     |     |     |
|     |     |     |     +----- day of week (0 - 6) (Sunday=0)
|     |     |     +------- month (1 - 12)
|     |     +--------- day of month (1 - 31)
|     +----------- hour (0 - 23)
+------------- min (0 - 59)


Rsync Remote Backup

This page Cygwin_Rsync details how to backup a Windows XP machine to another Windows XP machine over the internet. (using SSH + Rsync + Cygwin)


Wireshark / tcpflow

Tcpflow is a good command for putting tcp sessions back together, the following command will do this and remove the report.xml

tcpflow -T%t-%A-%a -x netviz -i [interface] -S enable_report=NO -o [output_dir] [wireshark collection filter]

TCP Replay

Will resend udp and tcp packets

N.B tcpreplay must send packets out an interface (cant send to the host its running on, so two VM's required)

sudo apt install tcpreplay


tcprewrite --infile=original.cap --outfile=changed.cap --srcipmap=0.0.0.0/0:<MY HOST IP>/32 --dstipmap=0.0.0.0/0:<MY HOST IP>/32 --enet-smac=<enp0s25 mac addr> --enet-dmac=<enp0s25 mac addr> --fixcsum

sudo tcpreplay --intf1=ens33  changed.cap

Control Characters

grep '[[:cntrl:]]' /tmp/file.txt

X Windows

A helpful guide www.redwireservices.com/remote-x11-for-linux-unix


RHEL 6

Unlock an account

pam_tally2 --user=username --reset

Sticky Bits

https://www.liquidweb.com/kb/how-do-i-set-up-setuid-setgid-and-sticky-bits-on-linux/

Dig

  1. get the address(es) for yahoo.com
dig yahoo.com A +noall +answer
  1. get a list of yahoo's mail servers
dig yahoo.com MX +noall +answer
  1. get a list of DNS servers authoritative for yahoo.com
dig yahoo.com NS +noall +answer
  1. get all of the above
❯ dig wiretap.nz ANY +noall +answer

; <<>> DiG 9.10.6 <<>> wiretap.nz ANY +noall +answer
;; global options: +cmd
wiretap.nz.		24	IN	A	172.67.133.52
wiretap.nz.		24	IN	A	104.21.13.215
wiretap.nz.		112	IN	AAAA	2606:4700:3030::6815:dd7
wiretap.nz.		112	IN	AAAA	2606:4700:3031::ac43:8534
  1. More obscurely, for the present anyway, you can also poll for a host’s IPv6 address using the AAAA option.
dig www.isc.org AAAA +short
  1. If the domain you want to query allows DNS transfers, you can get those, too. The reality of life on the Internet, however, is that very few domains allow unrestricted transfers these days.
dig yourdomain.com AXFR


Hex to Decimal

printf "%d\n", 0xFF

Decimal to Hex

printf "%02X\n", 255

Json to csv

brew install jq
cat blah.json | jq '.[]| join(",")'
cat outputUnquoted.json | jq -r '.[]| join(",")'

VI VIM

Go to end of file

G  (shift-g)

Vi line numbers

vim line numbers

:set nu

make a 2GB GPT Partition

Check which disk has no partitions

[root@serverb ~]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda    252:0    0   10G  0 disk
├─vda1 252:1    0    1M  0 part
├─vda2 252:2    0  200M  0 part /boot/efi
├─vda3 252:3    0  500M  0 part /boot
└─vda4 252:4    0  9.3G  0 part /
vdb    252:16   0    5G  0 disk
vdc    252:32   0    5G  0 disk
vdd    252:48   0    5G  0 disk

Check the disk has no label

[root@serverb ~]# parted /dev/vdb print
Error: /dev/vdb: unrecognised disk label
Model: Virtio Block Device (virtblk)
Disk /dev/vdb: 5369MB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags

Define GPT Partitioning scheme

[root@serverb ~]# parted /dev/vdb mklabel gpt
Information: You may need to update /etc/fstab.

Create 2GB Partition with xfs file system - start at sector 2048

[root@serverb ~]# parted /dev/vdb mkpart backup xfs 2048s 2GB
Information: You may need to update /etc/fstab.

Confirmation

[root@serverb ~]# parted /dev/vdb print
Model: Virtio Block Device (virtblk)
Disk /dev/vdb: 5369MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name    Flags
 1      1049kB  2000MB  1999MB               backup


Run the `udevadm settle` command. This command waits for the system to detect the new partition and to create the /dev/vdb1 device file.

udevadm settle


Format the partition

mkfs.xfs /dev/vdb1

Find UUID

lsblk --fs /dev/vdb1

Add to fstab

UUID=f74ed805-b1fc-401a-a5ee-140f97c6757d   /backup   xfs   defaults   0 0

reload systemctl

systemctl daemon-reload

Manually mount

mount /backup