How to Process a File Line by Line

Sometimes you need to process a file line by line. You can do this as follows:
 

First you need a file. Here is one I made earlier:
 

Linux > cat input_file
0123456789
9876543210
ABCDEFGHIJ
KLMNOPQRST
Linux >

 
I wrote this script to process the file I had created:
 
Linux > cat script.ksh
#!/bin/bash
i=1
for j in `cat input_file`
do
echo "Line $i = $j"
((i+=1))
done
Linux >

 
It produced the following output when I ran it:
 
Linux > ./script.ksh
Line 1 = 0123456789

Line 2 = 9876543210

Line 3 = ABCDEFGHIJ

Line 4 = KLMNOPQRST

Linux >

How to Add Text at the End of Every Line in vi

I received a file of SQL statements from a 3rd party but they arrived without a semi-colon at the end of each line so I could not run them. Here are the first few (there were nearly 10,000 in all):

delete from crm7.UDEFFIELD where UDEFFIELD_Id = 10043
delete from crm7.UDEFFIELD where UDEFFIELD_Id = 14606
delete from crm7.UDEFFIELD where UDEFFIELD_Id = 2278
delete from crm7.UDEFFIELD where UDEFFIELD_Id = 2631
delete from crm7.UDEFFIELD where UDEFFIELD_Id = 2515
delete from crm7.UDEFFIELD where UDEFFIELD_Id = 2259
delete from crm7.UDEFFIELD where UDEFFIELD_Id = 12327
delete from crm7.UDEFFIELD where UDEFFIELD_Id = 2142
delete from crm7.UDEFFIELD where UDEFFIELD_Id = 11344
delete from crm7.UDEFFIELD where UDEFFIELD_Id = 10339

I used vi to add them as follows:
(1) I pressed the Esc key.
(2) I typed the following command: :%s/$/;/
(3) I pressed Return.
This appended a semi-colon at the end of every line:
 
delete from crm7.UDEFFIELD where UDEFFIELD_Id = 10043;
delete from crm7.UDEFFIELD where UDEFFIELD_Id = 14606;
delete from crm7.UDEFFIELD where UDEFFIELD_Id = 2278;
delete from crm7.UDEFFIELD where UDEFFIELD_Id = 2631;
delete from crm7.UDEFFIELD where UDEFFIELD_Id = 2515;
delete from crm7.UDEFFIELD where UDEFFIELD_Id = 2259;
delete from crm7.UDEFFIELD where UDEFFIELD_Id = 12327;
delete from crm7.UDEFFIELD where UDEFFIELD_Id = 2142;
delete from crm7.UDEFFIELD where UDEFFIELD_Id = 11344;
delete from crm7.UDEFFIELD where UDEFFIELD_Id = 10339;

How to Change Your Command Prompt in UNIX or Linux

If you do not like your command prompt in UNIX or Linux, you can change it with the export PS1= command. In the example below, I started off with a command prompt of UBUNTU > :

UBUNTU > 

... then I changed it to show the name of the directory I was in. Here is one way to do this:

UBUNTU > export PS1='`pwd` > '
/home/andrew > cd Java
/home/andrew/Java > cd /
/ >

... and here is another:

/ > export PS1='$PWD > '

/ > cd

/home/andrew > cd Java

/home/andrew/Java > 

Finally, I changed it to say Andrew's PC: 

/home/andrew/Java > export PS1="Andrew's PC: "

Andrew's PC:

prstat -Z

If you have a Solaris server split up into zones, you can log on to the global zone and use prstat –Z. This produces a report like the one below. The last few lines show how much memory and CPU is being used by each zone. As usual, click on the image, if necessary, to enlarge it and bring it into focus:

How to See your IP Address in UBUNTU

You can do this from the Linux prompt as follows. The value you want is in bold. I hope to be using this a lot in future posts:

UBUNTU: ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0c:6e:e7:f1:92 
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:19 Base address:0x9800

lo        Link encap:Local Loopback 
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:2102 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2102 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:217564 (217.5 KB)  TX bytes:217564 (217.5 KB)

wlan0     Link encap:Ethernet  HWaddr 00:14:6c:8a:f4:72 
          inet addr:192.168.2.2  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: fe80::214:6cff:fe8a:f472/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:39507 errors:0 dropped:0 overruns:0 frame:0
          TX packets:37777 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:39342715 (39.3 MB)  TX bytes:7605863 (7.6 MB)

UBUNTU:

A Problem We Had With cron

We had a problem recently where a job was scheduled on cron something like this:

 

25 14 23 09 1 /bin/date > /usr/users/oracle/andrew/date.log 2>&1

 

This should run at 1425 hours but will it run just on Monday 23^rd September or will it run every Monday? I didn’t know as I only ever specify the day and month OR the day of the week, never both. I waited until the following time:

 

Tru64 > date

Mon Sep 30 14:20:40 BST 2013

Tru64 >

 

I added the line above to the cron file then waited until the following time:

 

Tru64 > date

Mon Sep 30 14:27:32 BST 2013

Tru64 >

 

The job had run as the output file had been updated with the run time so it seems that, on Tru64 at least, it will run every Monday irrespective of the day and month:

 

Tru64 > pwd

/usr/users/oracle/andrew

Tru64 > cat date.log

Mon Sep 30 14:25:00 BST 2013

Tru64 >

 

If you want it to run only on 23^rd September, you should schedule it like this:

 

25 14 23 09 * /bin/date > /usr/users/oracle/andrew/date.log 2>&1

 

I moved to a Solaris machine and added the line below to the cron file.

 

16 15 30 09 2 /usr/bin/date > /export/home/oracle/andrew/date.log 2>&1

 

This should run at 1516 hours but will it run just on Tuesday 30^th September or will it run every 30^th September? I waited until the following time:

 

Solaris > date

Monday, 30 September 2013 15:16:22 BST

Solaris >

 

The job had run as the output file had been updated with the run time so it seems that, on Solaris too, it will run every 30^th September irrespective of the day of the week:

 

Solaris > pwd

/export/home/oracle/andrew

Solaris > cat date.log

Mon Sep 30 15:16:00 BST 2013

Solaris >

 

If you want it to run only on Tuesdays, you should schedule it like this:

 

16 15 * * 2 /usr/bin/date > /export/home/oracle/andrew/date.log 2>&1

 

To summarise, if you schedule a job on cron and specify a day and month AND a day of the week, it will run when EITHER condition is satisfied, it does not need to wait until both conditions are satisfied.

UNIX compress Command

You can use the UNIX compress command to make a file smaller. Here is a worked example. I started by making a file called directory_list. I copied this file into another file called directory_list_save for future reference. Then I compressed the file called directory_list. This made the file much smaller and gave it a suffix of .Z. Finally, I compared the sizes of directory_list_save and directory_list.Z:

Tru64:/usr/users/oracle/andrew > ls -R $ORACLE_HOME > directory_list

Tru64:/usr/users/oracle/andrew > cp directory_list directory_list_save

Tru64:/usr/users/oracle/andrew > ls -l directory_list*

-rw-r--r--   1 oracle   dba      1971685 Feb  5 14:43 directory_list

-rw-r--r--   1 oracle   dba      1971685 Feb  5 14:43 directory_list_save

Tru64:/usr/users/oracle/andrew > compress directory_list

Tru64:/usr/users/oracle/andrew > ls -l directory_list*

-rw-r--r--   1 oracle   dba       558245 Feb  5 14:43 directory_list.Z

-rw-r--r--   1 oracle   dba      1971685 Feb  5 14:43 directory_list_save

Tru64:/usr/users/oracle/andrew >

You cannot read a file once you have compressed it as it is then in a different format. The purpose of compressing a file is to make it smaller so that it is easier to store or to send to somewhere else.  I created the file on a Tru64 machine then decided to send it to a Solaris machine using scp.

 

Tru64:/usr/users/oracle/andrew > scp directory_list.Z \

> zge-mktred-ddb1:/export/home/oracle/andrew

The authenticity of host 'zge-mktred-ddb1 (10.80.1.116)' can't be established.

RSA key fingerprint is d6:59:ba:8d:c6:24:21:1b:91:c2:3b:c5:5a:5b:8c:f9.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'zge-mktred-ddb1,10.80.1.116' (RSA) to the list of known hosts.

Password:

directory_list.Z                                                                                                           100%  545KB 545.2KB/s   00:00

Tru64:/usr/users/oracle/andrew >

I read somewhere that you could compress a file on one platform and uncompress it on another. So, once it was on the Solaris machine, I uncompressed it. This proved that a file compressed on Tru64 could be uncompressed on Solaris. Notice how this removed the .Z suffix from the file name. Finally, I compressed the file again on the Solaris machine:

 

Solaris:/export/home/oracle/andrew > ls -l directory_list*

-rw-r--r--   1 oracle   dba       558245 Feb  5 14:49 directory_list.Z

Solaris:/export/home/oracle/andrew > uncompress directory_list.Z

Solaris:/export/home/oracle/andrew > ls -l directory_list*

-rw-r--r--   1 oracle   dba      1971685 Feb  5 14:49 directory_list

Solaris:/export/home/oracle/andrew > compress directory_list

Solaris:/export/home/oracle/andrew > ls -l directory_list*

-rw-r--r--   1 oracle   dba       558245 Feb  5 14:49 directory_list.Z

Solaris:/export/home/oracle/andrew >

Then I sent the compressed file back to the Tru64 machine:

Solaris:/export/home/oracle/andrew > scp directory_list.Z \

> sredsvr8:/usr/users/oracle/andrew

The authenticity of host 'sredsvr8 (10.80.1.32)' can't be established.

DSA key fingerprint is 10:c8:14:df:7a:bb:a7:56:1d:7d:f5:c7:76:ee:2a:33.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'sredsvr8,10.80.1.32' (DSA) to the list of known hosts.

oracle@sredsvr8's password:

directory_list.Z     100% |**********************************************************************************************************|   545 KB    00:00

Solaris:/export/home/oracle/andrew >

Once the file was back on the Tru64 machine, I uncompressed it to prove that you could compress a file on a Solaris machine then uncompress it on a Tru64 machine. Finally I used the diff command to check that the uncompressed file matched the backup copy, which I made at the start of the example:

 

Tru64:/usr/users/oracle/andrew > ls -l directory_list*

-rw-r--r--   1 oracle   dba       558245 Feb  5 15:15 directory_list.Z

-rw-r--r--   1 oracle   dba      1971685 Feb  5 14:43 directory_list_save

Tru64:/usr/users/oracle/andrew > uncompress directory_list.Z

Tru64:/usr/users/oracle/andrew > ls -l directory_list*

-rw-r--r--   1 oracle   dba      1971685 Feb  5 15:15 directory_list

-rw-r--r--   1 oracle   dba      1971685 Feb  5 14:43 directory_list_save

Tru64:/usr/users/oracle/andrew > diff directory_list directory_list_save

Tru64:/usr/users/oracle/andrew >

The UNIX and Linux banner Command

With the banner command, you can use the # symbol to create large messages:

UBUNTU > banner Andrew
   #
  # #    #    #  #####   #####   ######  #    #
 #   #   ##   #  #    #  #    #  #       #    #
#     #  # #  #  #    #  #    #  #####   #    #
#######  #  # #  #    #  #####   #       # ## #
#     #  #   ##  #    #  #   #   #       ##  ##
#     #  #    #  #####   #    #  ######  #    #

UBUNTU > 

If the message has two words, the second appears below the first:

UBUNTU > banner Red Rum
######
#     #  ######  #####
#     #  #       #    #
######   #####   #    #
#   #    #       #    #
#    #   #       #    #
#     #  ######  #####

######
#     #  #    #  #    #
#     #  #    #  ##  ##
######   #    #  # ## #
#   #    #    #  #    #
#    #   #    #  #    #
#     #   ####   #    #

UBUNTU >

But, if you put the message in quotes, the two words appear on the same line:

UBUNTU > banner 'Red Rum'
######                          ######
#     #  ######  #####          #     #  #    #  #    #
#     #  #       #    #         #     #  #    #  ##  ##
######   #####   #    #         ######   #    #  # ## #
#   #    #       #    #         #   #    #    #  #    #
#    #   #       #    #         #    #   #    #  #    #
#     #  ######  #####          #     #   ####   #    #

UBUNTU >

The UNIX pwd Command

With the pwd command, you can see the name of the directory you are currently using: 

[andrew@localhost ~]$ cd /home/andrew

[andrew@localhost ~]$ pwd

/home/andrew

[andrew@localhost ~]$

chown

You can change the owner of a file with the chown command. Here is a file:

UBUNTU > ls -l
total 0
-rw-rw-r-- 1 andrew users 0 Aug 22 22:09 file1
UBUNTU >

The file is owned by andrew. If you want to change the owner to fred, you can do so as follows:

UBUNTU > sudo chown fred file1
UBUNTU > ls -l
total 0
-rw-rw-r-- 1 fred users 0 Aug 22 22:09 file1
UBUNTU >

How to Create a File of a Particular Size in UNIX or Linux

I needed a file with a size of 512 bytes for an Oracle test. I created it as follows:

UBUNTU > dd if=/dev/zero of=test1 bs=1 count=512
512+0 records in
512+0 records out
512 bytes (512 B) copied, 0.0140867 s, 36.3 kB/s
UBUNTU > ls -l test1
-rw-rw-r-- 1 andrew andrew 512 Aug 11 22:34 test1
UBUNTU >

To create a file with a size of 1000 bytes, you only have to change count=512 to count=1000:

UBUNTU > dd if=/dev/zero of=test2 bs=1 count=1000
1000+0 records in
1000+0 records out
1000 bytes (1.0 kB) copied, 0.0174102 s, 57.4 kB/s
UBUNTU > ls -l test2
-rw-rw-r-- 1 andrew andrew 1000 Aug 11 22:35 test2
UBUNTU >

How to Tell if a UNIX User is Locked

You can check if a UNIX user is locked with the passwd -s command, followed by the username. If the letters LK appear at the start of the output, after the username, the user is locked:

Solaris root user > passwd -s oracle
oracle    LK    06/15/12     7    56     7
Solaris root user >

You can unlock a user with the passwd -u command, followed by the username:

Solaris root user > passwd -u oracle
passwd: password information changed for oracle
Solaris root user>

After unlocking the user, the letters PS replace the letters LK in the output of the passwd -s command. This simply means that the user has a password:

Solaris root user > passwd -s oracle
oracle    PS    06/15/12     7    56     7
Solaris root user >

In Spanish / en español

Examples Using wc

Here is a file:

Linux > cat file1
Increase productivity across your entire Oracle stack
with access to unique on-line tools and support
communities - all part of your Oracle Premier Support
service.
Use Oracle proactive support to reduce risks and
lower your organisations costs through preventive
maintenance. Benefit from health services, intelligent
fault management and timely, personalized advice,
backed by Oracle support experts.
Linux >

You can count the number of lines, words and characters in the file like this:

Linux > wc file1
      9      56     403 file1
Linux >

If you only need the number of lines, here is what you must do:

Linux > wc -l file1
      9 file1
Linux >

The following example counts the words in the file:

Linux > wc -w file1
     56 file1
Linux >

With wc -c you can count the number of characters in the file:

Linux > wc -c file1
    403 file1
Linux >

You can find the length of the longest line like this:

Linux > wc -L file1
     54 file1
Linux >

Here is another file:

Linux > cat file2
The 2012 threat landscape is one of constant and rapid
change.
Where new website vulnerabilities are tirelessly sought
and exploited.
Where new social engineering techniques can increasingly
circumvent external security measures and infect from
within.
And where hackers now employ multiple attack
methodologies and vectors to penetrate defences and
steal information.
Linux >

You can give more than one filename as follows:

Linux > wc -lc file1 file2
      9     403 file1
     10     369 file2
     19     772 total
Linux >

In the next example, cat is used to see the result without the filename:

Linux > cat file1|wc -L
     54
Linux >

If you give the name of a file which does not exist, wc replies with an appropriate error message:

Linux > wc file3
wc: file3: No such file or directory
Linux >

Case Sensitivity in UNIX and Linux

In UNIX and Linux, names are case sensitive. Here is an example. We start in an empty directory. Then we create 4 files with the same name. One filename is in lower case. One file name is in upper case. The other filenames have a combination of upper and lower case letters: 

UBUNTU > ls

UBUNTU > touch fila1 Fila1 FiLa1 FILA1

UBUNTU > ls

fila1 Fila1 FiLa1 FILA1

UBUNTU > 

You can do the same thing with directories:

UBUNTU > ls -l

total 0
-rw-r--r-- 1 andrew users 0 Aug 27 20:54 fila1
-rw-r--r-- 1 andrew users 0 Aug 27 20:54 Fila1
-rw-r--r-- 1 andrew users 0 Aug 27 20:54 FiLa1
-rw-r--r-- 1 andrew users 0 Aug 27 20:54 FILA1
UBUNTU > mkdir dir1 Dir1 DiR1 DIR1
UBUNTU > ls -l
total 16
drwxr-xr-x 2 andrew users 4096 Aug 27 21:00 dir1
drwxr-xr-x 2 andrew users 4096 Aug 27 21:00 Dir1
drwxr-xr-x 2 andrew users 4096 Aug 27 21:00 DiR1
drwxr-xr-x 2 andrew users 4096 Aug 27 21:00 DIR1
-rw-r--r-- 1 andrew users    0 Aug 27 20:54 fila1
-rw-r--r-- 1 andrew users    0 Aug 27 20:54 Fila1
-rw-r--r-- 1 andrew users    0 Aug 27 20:54 FiLa1
-rw-r--r-- 1 andrew users    0 Aug 27 20:54 FILA1
UBUNTU >

Disk Speed

You can use the following command to see the speed of a hard disk: 

UBUNTU > time dd if=/dev/zero of=large_file \

> bs=4096 count=2000000

2000000+0 records in

2000000+0 records out

8192000000 bytes (8.2 GB) copied, 192.95 s, 42.5 MB/s

real 3m13.102s

user 0m6.148s

sys 1m29.718s

UBUNTU >

hostname and uname -n

You can see the name of the machine you are using with the hostname command:

Linux > hostname
LUBUNTU-Laptop
Linux >

The uname -n command does the same thing:

Linux > uname -n
LUBUNTU-Laptop
Linux >