Difference between revisions of "Unix"

From neil.tappsville.com
Jump to navigationJump to search
Line 365: Line 365:
  
 
====TCP Replay====
 
====TCP Replay====
 +
 +
Will resend udp and tcp packets
  
 
N.B tcpreplay must send packets out an interface (cant send to the host its running on, so two VM's required)
 
N.B tcpreplay must send packets out an interface (cant send to the host its running on, so two VM's required)

Revision as of 21:25, 13 October 2019

General UNIX How-to's

Find when the latest sub-directory was created given a path and search pattern (aka datachannel done files..)

date; bash -c 'for DIRECTORY in `find /appl/proviso/data/datachannel/ -name done -type d -user pvuser`; do echo "==";echo "$DIRECTORY"; ls -rt $DIRECTORY | tail -1 ;  done'

Count the files waiting to be processed

date; find /appl/proviso/data/datachannel/BLB*/output  | cut -d "/" -f 6 | sort | uniq -c

Run shell command every x seconds

bash -c 'while [[ 0 ]]; do echo "----"; date '+%H:%M:%S'; ls -l /appl/proviso/datachannel/BLB.2.251/output; echo "----"; sleep 5; done'

grep with OR

egrep -e 'JMS''PROCESS|SAM''CONNECT'

For the locker below - better to use extended ps output -incase the filename gets truncated (works on spac)

/usr/ucb/ps -xwww

how to get the full path to the executable


% ps -ef | grep [[processname]]
root     24466  0.0  0.0   1476   280 ?        S     2009   0:00 supervise sshd
% sudo ls -l /proc/24466/exe


Bash script to force locking / only one instance of a script will run at a time

#!/bin/bash
#
# watchdog
#
# Run as a cron job to keep an eye on what_to_monitor which should always
# be running. Restart what_to_monitor and send notification as needed.
#

NAME=dc_log_sam_blb_errors.pl
ENV=/appl/proviso/datachannel/dataChannel.env
APATH=/appl/proviso/Application/ExtAdapter/SAM_MONITOR
PARAM=/appl/proviso/datachannel/log/proviso.log
GREP=/bin/grep
PS=/bin/ps

$PS -ef|$GREP -v grep|$GREP $NAME >/dev/null 2>&1
case "$?" in
 0)
 # It is running in this case so we do nothing.
 ;;
 1)
 echo "$NAME is NOT RUNNING. Starting $NAME"
 . $ENV
 cd $APATH
 ./$NAME $PARAM 2>&1 >/dev/null &
 ;;
esac

exit

Remove files EDS put in temp and never remove (cronjob as root)

## remove old EDS files from malformed scripts that run
00 05 * * * find /tmp/CTSA* -user root -ctime +2 -type f -exec rm -f {} \; 2>&1
00 05 * * * find /tmp/disktacct* -user root -ctime +2 -type f -exec rm -f {} \; 2>&1
00 05 * * * find /tmp/locked* -user root -ctime +2 -type f -exec rm -f {} \; 2>&1

Keep a process running after you log off - with the right flags and run as root a process can never be stopped! Cli output will be written to nohup.out in the directory you run the command from

nohup <command> &


Defunct Processes - Find and map their parents

/usr/proc/bin/ptree `ps -ef | grep -v ^\ \ \ \ root | awk '/<defunct>/ {print $2}'` | less

Symbolic Link

ln -s {/path/to/file-name} {link-name}
rm {link-name}


Send a text file via email from unix

cat <file> | mailx -s 'sunject' <email@dot.com>

Start Apache (if some retard didnt create the init.d file)

/usr/local/apache2/bin/httpd -k start

Using zip instead of tar (for files with REALLY long names) will output myzipfile.zip containing all data in directory (within directory folder in the zip)

zip -r myzipfile directory


Find the total size of all files in this folder like abc

du -ks /appl/proviso/dataload/log/*.pcap | awk '{t+=$1}END{printf "%.2f Mb\n", t/1024}'

Remember to escape the $ " and  \

 ./run_command_on_dataloaders.sh " du -ks /appl/proviso/dataload/log/*.pcap | awk '{t+=\$1}END{printf \"%.2f Mb\", t/1024}' && ls -l /appl/proviso/dataload/log/*.pcap | wc -l"

Log Memory Changes for a Process run with NCO_MEM.sh PID

#! /bin/sh
###############################################
# Script to log memory usage over time
###############################################
export PLATFORM
PLATFORM=`uname -p`
###############################################
# Edit SLEEP as required
###############################################
SLEEP=10
#
if [ $# -eq 1 ]
then
PID=$1
else
echo "Usage : `basename $0` [PID}"
echo "SLEEP=$SLEEP"
exit
fi
export DATE PS PID SLEEP OLD_MEM MEM OLD_VMEM VMEM OLD_RSS RSS
echo "PLATFORM=$PLATFORM"
###############################################
###############################################
if [ "$PLATFORM" = "" ]
then
echo "Unsupported platform"
exit
###############################################
# LINUX
###############################################
elif [ "$PLATFORM" = "x86_64" ]
then
PS=`ps -p $PID -o "size rss vsz pmem time pcpu" | head -1`
date '+%d/%m/%y %H:%M:%S'
echo "Date         : $PS"
OLD_MEM=0
OLD_VMEM=0
OLD_RSS=0
while true
do
MEM=`ps -p $PID -o size | tail -1`
VMEM=`ps -p $PID -o vsz | tail -1`
RSS=`ps -p $PID -o rss | tail -1`
if [ $MEM -ne $OLD_MEM -o $VMEM -ne $OLD_VMEM -o $RSS -ne $OLD_RSS ]
then
OLD_VMEM=$VMEM
OLD_MEM=$MEM
OLD_RSS=$RSS
DATE=`date '+%y%m%d%H%M%S'`
PS=`ps -p $PID -o "size rss vsz pmem time pcpu" | tail -1`
echo "$DATE : $PS"
fi
sleep $SLEEP
done
###############################################
# Solaris
###############################################
elif [ "$PLATFORM" = "sparc" ]
then
PS=`ps -p $PID -o "osz rss vsz pmem time pcpu" | head -1`
date '+%d/%m/%y %H:%M:%S'
echo "Date         : $PS"
OLD_MEM=0
OLD_VMEM=0
OLD_RSS=0
while true
do
MEM=`ps -p $PID -o osz | tail -1`
VMEM=`ps -p $PID -o vsz | tail -1`
RSS=`ps -p $PID -o rss | tail -1`
if [ $MEM -ne $OLD_MEM -o $VMEM -ne $OLD_VMEM -o $RSS -ne $OLD_RSS ]
then
OLD_VMEM=$VMEM
OLD_MEM=$MEM
OLD_RSS=$RSS
DATE=`date '+%y%m%d%H%M%S'`
DATE=`date '+%d/%m/%y %H:%M:%S'`
PS=`ps -p $PID -o "osz rss vsz pmem time pcpu" | tail -1`
echo "$DATE : $PS"
fi
sleep $SLEEP
done
###############################################
# AIX
###############################################
elif [ "$PLATFORM" = "powerpc" ]
then
#PS=`ps -p $PID -o "osz rss vsz pmem time pcpu" | head -
PS=`ps -p $PID -o "rssize vsz pmem" | head -1`
date '+%d/%m/%y %H:%M:%S'
echo "Date         : $PS"
OLD_MEM=0
OLD_VMEM=0
while true
do
MEM=`ps -p $PID -o rssize | tail -1`
VMEM=`ps -p $PID -o vsz | tail -1`
if [ $MEM -ne $OLD_MEM -o $VMEM -ne $OLD_VMEM ]
then
OLD_VMEM=$VMEM
OLD_MEM=$MEM
DATE=`date '+%y%m%d%H%M%S'`
DATE=`date '+%d/%m/%y %H:%M:%S'`
PS=`ps -p $PID -o "rssize vsz pmem" | tail -1`
echo "$DATE : $PS"
fi
sleep $SLEEP
done
###############################################
# Last FI
###############################################
fi
#EOF


Format the xml files in the current directory

bash -c 'for XML in `find *.xml`; do echo "$XML"; xmllint -format $XML > $XML.formatted; done'


SED and REGEX on the command line

cat blah | cut -d " " -f4 | sed 's/[0-9]*//g'

www.tutorialspoint.com/unix/unix-regular-expressions.htm


CUT and CUT -ordered

Normal cut will seperate on delimiter -d ";" and print fields -f 1,2-10 quickly but will only do them in order.

To be able to specify the output in a specific order use awk

awk -F";" '{print $9,$24,$34,$4,$11,$12,$16,$10,$13,$14,$15,$23,$33,$5,$6,$7,$19,$20,$8,$22,$36,$26,$35,$25,$1,$2,$17,$18,$27,$28,$29,$30,$32,$31,$21,$37,$38,$3}' properties_new.csv

remove a file with a strange name

$ ls -rlt
total 2018
drwxrwxrwx   2 pvuser   proviso      512 Jan 23 14:56 old
-rw-r-----   1 pvuser   proviso  1018744 Jan 23 16:03 OSS_Inventory.csv
-rw-rw-rw-   1 pvuser   proviso        0 Jan 23 17:04 --?
-rw-rw-rw-   1 pvuser   proviso        0 Jan 23 17:04 -help
$ rm -- --?
$ ls -l
total 2018
-rw-rw-rw-   1 pvuser   proviso        0 Jan 23 17:04 -help
-rw-r-----   1 pvuser   proviso  1018744 Jan 23 16:03 OSS_Inventory.csv
drwxrwxrwx   2 pvuser   proviso      512 Jan 23 14:56 old


Find open files on a partition

lsof will list open file and sockets

lsof

If lsof is not installed - or you dont have rights..

to find active processes for a filesystem do "fuser -c filesystem"
[sf2395:root]:/appl/proviso/log > fuser -c /appl/proviso/log
/appl/proviso/log:    25190c   21255c   24996c   18286o    6915o    5473o    4816o
[sf2395:root]:/appl/proviso/log >


then do ptree of the pid above to see what they're doing:
[sf2395:root]:/appl/proviso/log > ptree 18286
1440  /usr/sbin/cron
  18151 sh -c . /appl/proviso/DataAccessAPI/mpoi/edr/run.sh > /dev/null 2>&1
    18286 /appl/proviso/DataAccessAPI/j2sdk1.4.1_06/bin/java -classpath .:/appl/proviso/D
[sf2395:root]:/appl/proviso/log >


Netcat - pipe a unix input to another machines socket - or listen to a socket and pipe the output.

www.tutorialspoint.com/unix_commands/nc.htm


www.g-loaded.eu/2006/11/06/netcat-a-couple-of-useful-examples/


nc - arbitrary TCP and UDP connections and listens


* nc  -words [-46Ddhklnr StUuvz] [-i interval] [-p source'port] [-s source'ip'address] [-T To S] [-w timeout] [-X proxy'protocol] [Xo -x proxy_address[: port]] [hostname] [port[s]]

TCP Server / Listener
nc -n <port>
TCP Client
nc -n <ipaddress> <port>

some OS you must specify port with -p (thanks centos)


DESCRIPTION The nc (or netcat) utility is used for just about anything under the sun involving TCP or UDP. It can open TCP connections, send UDP packets, listen on arbitrary TCP and UDP ports, do port scanning, and deal with both IPv4 and IPv6. Unlike telnet(1), nc scripts nicely, and separates error messages onto standard error instead of sending them to standard output, as telnet(1) does with some.

Common uses include 
simple TCP proxies

shell-script based HTTP clients and servers network daemon testing a SOCKS or HTTP Proxy Command for ssh(1) and much, much more


Crontab Syntax

Im always forgetting the day of week part

*     *     *     *     *  command to be executed
-     -     -     -     -
|     |     |     |     |
|     |     |     |     +----- day of week (0 - 6) (Sunday=0)
|     |     |     +------- month (1 - 12)
|     |     +--------- day of month (1 - 31)
|     +----------- hour (0 - 23)
+------------- min (0 - 59)


Rsync Remote Backup

This page Cygwin_Rsync details how to backup a Windows XP machine to another Windows XP machine over the internet. (using SSH + Rsync + Cygwin)


Wireshark / tcpflow

Tcpflow is a good command for putting tcp sessions back together, the following command will do this and remove the report.xml

tcpflow -T%t-%A-%a -x netviz -i [interface] -S enable_report=NO -o [output_dir] [wireshark collection filter]

TCP Replay

Will resend udp and tcp packets

N.B tcpreplay must send packets out an interface (cant send to the host its running on, so two VM's required)

sudo apt install tcpreplay


tcprewrite --infile=original.cap --outfile=changed.cap --srcipmap=0.0.0.0/0:<MY HOST IP>/32 --dstipmap=0.0.0.0/0:<MY HOST IP>/32 --enet-smac=<enp0s25 mac addr> --enet-dmac=<enp0s25 mac addr> --fixcsum

sudo tcpreplay --intf1=ens33  changed.cap

Control Characters

grep '[[:cntrl:]]' /tmp/file.txt

X Windows

A helpful guide www.redwireservices.com/remote-x11-for-linux-unix


RHEL 6

Unlock an account

pam_tally2 --user=username --reset