Need help with this for loop

Fireball3 · March 7, 2014

OK, so how would our code look like?

You can have the check_hour function set a global variable and ALSO return that via the return command.
Then you can have the best of both worlds.

A variable to check the last status of the call and a return code when needed.

I will always have to check if hours match.

So that call will always be necessary.

WeeboTech · March 7, 2014

Here's some example code.

#!/bin/bash

noCountdownHours="18 19 20 21 22 23"   

check_hour_list()
{
  HOUR=$(date +%H)
  HOUR_MATCHED_RC=1

  for CountdownHour in ${noCountdownHours}
  do  if [ ${HOUR} = ${CountdownHour} ] ; then
         HOUR_MATCHED_RC=0
         return ${HOUR_MATCHED_RC}
      fi
  done

  return ${HOUR_MATCHED_RC}
}

hour=$(date +%H)
hourMatch=$(echo "$noCountdownHours" | grep "$hour" | wc -l)
echo "noCountdownHours: ${noCountdownHours}, current hour: $hour, match? $hourMatch"

if check_hour_list
   then echo "check_hour_list current hour matched"
   else echo "check_hour_list current hour DID NOT MATCH"
fi

check_hour_list
if [ $? -eq 0 ]
   then echo "check_hour_list current hour matched"
   else echo "check_hour_list current hour DID NOT MATCH"
fi

check_hour_list
if [ ${HOUR_MATCHED_RC} -eq 0 ]
   then echo "check_hour_list current hour matched"
   else echo "check_hour_list current hour DID NOT MATCH"
fi

Test output

root@unRAID:/tmp# ./x.sh 
noCountdownHours: 18 19 20 21 22 23, current hour: 16, match? 0
check_hour_list current hour DID NOT MATCH
check_hour_list current hour DID NOT MATCH
check_hour_list current hour DID NOT MATCH

root@unRAID:/tmp# ./x.sh 
noCountdownHours: 16 18 19 20 21 22 23, current hour: 16, match? 1
check_hour_list current hour matched
check_hour_list current hour matched
check_hour_list current hour matched

Fireball3 · March 7, 2014

OK, now we have a variable set and don't need to call the function.

But how do we notice if the hour has passed and we need to check again?

Edit: wait - I did not scroll down in the code window...

Edit2: no, question still valid

By the way,

I rewrote the logger passage and now it works. The "-" was odd.

How can I log multiple lines:

log_message " ----------- $program_name version $version ---------------------
Configuration settings.
Power down instead of sleep: $powerDownInsteadOfSleep
Check HDD activity: $checkHDDs
Check TCP activity: $checkTCP
Check SSH connections: $checkSSH
Check TELNET connections: $checkTELNET
Check for online clients: $pingIPs
Check local logins: $checkSHELL
Check for lock file (tmp/nos3sleep.lock): $checkLockFile
No sleep/powerdown during hours: $noCountdownHours
Save SMART values before powerdown: $save_SMART
--- After wake up from sleep: ----
Renew DHCP: $doDhcpRenewal
Force GB LAN: $forceGb
-----------------------------------------------------------------"

I don't have line feed when doing it like this.

WeeboTech · March 7, 2014

I rewrote the logger passage and now it works. The "-" was odd.

How can I log multiple lines:


while read MESSAGE
do log_message "${MESSAGE}"
done <-EOF
----------- $program_name version $version ---------------------
Configuration settings.
Power down instead of sleep: $powerDownInsteadOfSleep
Check HDD activity: $checkHDDs
Check TCP activity: $checkTCP
Check SSH connections: $checkSSH
Check TELNET connections: $checkTELNET
Check for online clients: $pingIPs
Check local logins: $checkSHELL
Check for lock file (tmp/nos3sleep.lock): $checkLockFile
No sleep/powerdown during hours: $noCountdownHours
Save SMART values before powerdown: $save_SMART
--- After wake up from sleep: ----
Renew DHCP: $doDhcpRenewal
Force GB LAN: $forceGb
-----------------------------------------------------------------
EOF

It's not efficient doing it like that.

You could logger directly with

logger -t$program <-EOF
EOF

Fireball3 · March 7, 2014

It's not efficient doing it like that.

Yes, I thought that too, I have it like | logger -t...

What's this <-EOF for? (end of file)?

WeeboTech · March 7, 2014

It's not efficient doing it like that.

Yes, I thought that too, I have it like | logger -t...

What's this <-EOF for? (end of file)?

It's a HERE document.

http://tldp.org/LDP/abs/html/here-docs.html

Fireball3 · March 8, 2014

Here's some example code.

#!/bin/bash

noCountdownHours="18 19 20 21 22 23"   

check_hour_list()
{
  HOUR=$(date +%H)
  HOUR_MATCHED_RC=1

  for CountdownHour in ${noCountdownHours}
  do  if [ ${HOUR} = ${CountdownHour} ] ; then
         HOUR_MATCHED_RC=0
         return ${HOUR_MATCHED_RC}
      fi
  done

  return ${HOUR_MATCHED_RC}
}

hour=$(date +%H)
hourMatch=$(echo "$noCountdownHours" | grep "$hour" | wc -l)
echo "noCountdownHours: ${noCountdownHours}, current hour: $hour, match? $hourMatch"

if check_hour_list
   then echo "check_hour_list current hour matched"
   else echo "check_hour_list current hour DID NOT MATCH"
fi

check_hour_list
if [ $? -eq 0 ]
   then echo "check_hour_list current hour matched"
   else echo "check_hour_list current hour DID NOT MATCH"
fi

check_hour_list
if [ ${HOUR_MATCHED_RC} -eq 0 ]
   then echo "check_hour_list current hour matched"
   else echo "check_hour_list current hour DID NOT MATCH"
fi

OK, now we have a variable set and don't need to call the function.

But how do we notice if the hour has passed and we need to check again?

WeeboTech · March 8, 2014

Here's some example code.

#!/bin/bash

noCountdownHours="18 19 20 21 22 23"   

check_hour_list()
{
  HOUR=$(date +%H)
  HOUR_MATCHED_RC=1

  for CountdownHour in ${noCountdownHours}
  do  if [ ${HOUR} = ${CountdownHour} ] ; then
         HOUR_MATCHED_RC=0
         return ${HOUR_MATCHED_RC}
      fi
  done

  return ${HOUR_MATCHED_RC}
}

hour=$(date +%H)
hourMatch=$(echo "$noCountdownHours" | grep "$hour" | wc -l)
echo "noCountdownHours: ${noCountdownHours}, current hour: $hour, match? $hourMatch"

if check_hour_list
   then echo "check_hour_list current hour matched"
   else echo "check_hour_list current hour DID NOT MATCH"
fi

check_hour_list
if [ $? -eq 0 ]
   then echo "check_hour_list current hour matched"
   else echo "check_hour_list current hour DID NOT MATCH"
fi

check_hour_list
if [ ${HOUR_MATCHED_RC} -eq 0 ]
   then echo "check_hour_list current hour matched"
   else echo "check_hour_list current hour DID NOT MATCH"
fi

OK, now we have a variable set and don't need to call the function.

But how do we notice if the hour has passed and we need to check again?

Run the function again.

Fireball3 · March 8, 2014

Run the function again.

Then I'm here again - or not?:

Edit:
According to my findings this should be correct:

(command expansion for check_hour)
if [[ $(check_hour) -eq 0 && "$LockFile" -eq 0 ]]

WeeboTech · March 9, 2014

Run the function again.

Then I'm here again - or not?:
Edit:
According to my findings this should be correct:

(command expansion for check_hour)
if [[ $(check_hour) -eq 0 && "$LockFile" -eq 0 ]]

That will work. It does actually fork another process to get the value of the shell function.

it could be done a lil more awkwardly like

if check_hour_list && [ $lockfile -eq 0 ]
   then echo "check_hour_list current hour matched"
   else echo "check_hour_list current hour DID NOT MATCH"
fi

or

check_hour_list
if [[ $CHECK_HOUR_RC -eq 0 && $lockfile -eq 0 ]]
   then echo "check_hour_list current hour matched"
   else echo "check_hour_list current hour DID NOT MATCH"
fi

or

check_hour_list
if [[ $? -eq 0 && $lockfile -eq 0 ]]
   then echo "check_hour_list current hour matched"
   else echo "check_hour_list current hour DID NOT MATCH"
fi

if you are interested in the shell programming sub-process example, I can post it so you get visual of what happens.

I don't want to put more in here then you need. Let me know.

Fireball3 · March 9, 2014

Thank you, it's already overwhelming what you provided.

So after all, there is no other way then forking another process in this case - just for confirmation.

I will have to study the last posts and change that accordingly in my script.

And then I have to check if everything works.

I need to look into the debug function of the shell (set -x -v).

Seems useful from what I read.

Are you familiar with the DOS shell also?

Anyway, I will open another thread for that problem tomorrow.

WeeboTech · March 9, 2014

Thank you, it's already overwhelming what you provided.

So after all, there is no other way then forking another process in this case - just for confirmation.

I will have to study the last posts and change that accordingly in my script.

And then I have to check if everything works.

I need to look into the debug function of the shell (set -x -v).

Seems useful from what I read.

Are you familiar with the DOS shell also?

Anyway, I will open another thread for that problem tomorrow.

The 3 versions I've shown only fork to run the date command, other then that they do not fork a shell as a sub process.

whenever you do $(command) or `command` the process splits(forks) after a pipe is set up.

It's not normally an issue these days unless you are doing this in a loop many many times.

Example: if a file is being read in the shell for many lines and each line has a bunch of VAR=`cut` commands.

I had a shell program once checking a file's age via external call. When the file was small it ran in under a minute. As the control file got larger and larger the time to run exponentially grew until it was taking over 3 minutes to test all the files.

I re-wrote it in perl and the process now ran in under 3 seconds.

In the case for the autosleep there's probably little degradation.

I just saw an opportunity to make it more efficient and provide food for thought on an easy habit that will bite someone later on.

Fireball3 · March 9, 2014

The 3 versions I've shown only fork to run the date command, other then that they do not fork a shell as a sub process.
whenever you do $(command) or `command` the process splits(forks) after a pipe is set up.

If it's that easy and makes such a difference, mabe you show me that example you mentioned.

WeeboTech · March 9, 2014

here is my test script, you can comment uncomment as you want to test.

The goal was to test/show variances between usage syntax.

#!/bin/bash

noCountdownHours="16 18 19 20 21 22 23"

check_hour_list()
{
  HOUR=$(date +%H)
  HOUR_MATCHED_RC=1


  echo "check_hour's pid: $$ ppid: $PPID, BASHPID: $BASHPID">&2

  for CountdownHour in ${noCountdownHours}
  do  if [ ${HOUR} = ${CountdownHour} ] ; then 
         HOUR_MATCHED_RC=0
         return ${HOUR_MATCHED_RC}
      fi
  done

  return ${HOUR_MATCHED_RC}
}

hour=$(date +%H)
hourMatch=$(echo "$noCountdownHours" | grep "$hour" | wc -l)
echo "noCountdownHours: ${noCountdownHours}, current hour: $hour, match? $hourMatch"

if check_hour_list
   then echo "check_hour_list current hour matched"
   else echo "check_hour_list current hour DID NOT MATCH"
fi

check_hour_list
if [ $? -eq 0 ]
   then echo "check_hour_list current hour matched"
   else echo "check_hour_list current hour DID NOT MATCH"
fi

check_hour_list
if [ ${HOUR_MATCHED_RC} -eq 0 ]
   then echo "check_hour_list current hour matched"
   else echo "check_hour_list current hour DID NOT MATCH"
fi

if [[ $(check_hour_list) -eq 0 ]]
   then echo "check_hour_list current hour matched"
   else echo "check_hour_list current hour DID NOT MATCH"
fi

echo "check_hour_list: $(check_hour_list)" >&2

lockfile=0
if check_hour_list && [ $lockfile -eq 0 ]
   then echo "check_hour_list current hour matched"
   else echo "check_hour_list current hour DID NOT MATCH"
fi

check_hour_list
if [[ $? -eq 0 && $lockfile -eq 0 ]]
   then echo "check_hour_list current hour matched"
   else echo "check_hour_list current hour DID NOT MATCH"
fi

check_hour_list
if [[ ${HOUR_MATCHED_RC} -eq 0 && $lockfile -eq 0 ]]
   then echo "check_hour_list current hour matched"
   else echo "check_hour_list current hour DID NOT MATCH"
fi

exit 

let i=1
lockfile=0
while [[ $i -lt 10000 ]]
do    (( i++ ))
      # if check_hour_list
      #   then : echo -e "$i \c"
      #   else : echo -e "$i \c"
      # fi
      if [[ $(check_hour_list) -eq 0 ]]
         then : echo -e "$i \c"
         else : echo -e "$i \c"
      fi
done

First Run

root@unRAID:/boot/bin# ./check_hour.sh  
noCountdownHours: 16 18 19 20 21 22 23, current hour: 13, match? 0
check_hour's pid: 19006 ppid: 29047, BASHPID: 19006
check_hour_list current hour DID NOT MATCH
check_hour's pid: 19006 ppid: 29047, BASHPID: 19006
check_hour_list current hour DID NOT MATCH
check_hour's pid: 19006 ppid: 29047, BASHPID: 19006
check_hour_list current hour DID NOT MATCH

check_hour's pid: 19006 ppid: 29047, BASHPID: 19015
check_hour_list current hour matched
check_hour's pid: 19006 ppid: 29047, BASHPID: 19018

check_hour_list: 
check_hour's pid: 19006 ppid: 29047, BASHPID: 19006
check_hour_list current hour DID NOT MATCH
check_hour's pid: 19006 ppid: 29047, BASHPID: 19006
check_hour_list current hour DID NOT MATCH
check_hour's pid: 19006 ppid: 29047, BASHPID: 19006
check_hour_list current hour DID NOT MATCH

It can be seen that some syntax usage causes a fork when examining BASHPID.

Now to prove/disprove how this affects things we have to iterate many many times over the same code.

As I said before, in this particular usage and on an idle system it doesn't matter all that much.

When you have other jobs running it becomes even more apparent and the CPU load will spike rapidly.

This is a test of check_hour doing 1000 iterations with if [[ $(check_hour) ]]

root@unRAID:/boot/bin# time ./check_hour.sh 
noCountdownHours: 16 18 19 20 21 22 23, current hour: 13, match? 0
check_hour_list current hour DID NOT MATCH
check_hour_list current hour DID NOT MATCH
check_hour_list current hour DID NOT MATCH
check_hour_list current hour matched
check_hour_list: 
check_hour_list current hour DID NOT MATCH
check_hour_list current hour DID NOT MATCH
check_hour_list current hour DID NOT MATCH

real    0m8.335s
user    0m0.120s
sys     0m0.270s

And another doing if check_hour

root@unRAID:/boot/bin# time ./check_hour.sh 
noCountdownHours: 16 18 19 20 21 22 23, current hour: 13, match? 0
check_hour_list current hour DID NOT MATCH
check_hour_list current hour DID NOT MATCH
check_hour_list current hour DID NOT MATCH
check_hour_list current hour matched
check_hour_list: 
check_hour_list current hour DID NOT MATCH
check_hour_list current hour DID NOT MATCH
check_hour_list current hour DID NOT MATCH

real    0m6.884s
user    0m0.740s
sys     0m0.170s

small potatoes at the moment.

When we bump this up to 10,000 iterations.

let i=1
lockfile=0
while [[ $i -lt 10000 ]]
do    (( i++ ))
      # if check_hour_list
      #   then : echo -e "$i \c"
      #   else : echo -e "$i \c"
      # fi
      if [[ $(check_hour_list) -eq 0 ]]
         then : echo -e "$i \c"
         else : echo -e "$i \c"
      fi
doneroot@unRAID:/boot/bin# time ./check_hour.sh 
noCountdownHours: 16 18 19 20 21 22 23, current hour: 13, match? 0
check_hour_list current hour DID NOT MATCH
check_hour_list current hour DID NOT MATCH
check_hour_list current hour DID NOT MATCH
check_hour_list current hour matched
check_hour_list: 
check_hour_list current hour DID NOT MATCH
check_hour_list current hour DID NOT MATCH
check_hour_list current hour DID NOT MATCH

real    1m22.140s
user    0m1.550s
sys     0m1.930s

and if check_hours

let i=1
lockfile=0
while [[ $i -lt 10000 ]]
do    (( i++ ))
      if check_hour_list
         then : echo -e "$i \c"
         else : echo -e "$i \c"
      fi
      # if [[ $(check_hour_list) -eq 0 ]]
      #   then : echo -e "$i \c"
      #   else : echo -e "$i \c"
      # fi
done


root@unRAID:/boot/bin# time ./check_hour.sh 
noCountdownHours: 16 18 19 20 21 22 23, current hour: 13, match? 0
check_hour_list current hour DID NOT MATCH
check_hour_list current hour DID NOT MATCH
check_hour_list current hour DID NOT MATCH
check_hour_list current hour matched
check_hour_list: 
check_hour_list current hour DID NOT MATCH
check_hour_list current hour DID NOT MATCH
check_hour_list current hour DID NOT MATCH

real    1m8.321s
user    0m6.870s
sys     0m2.020s

Again in this example, small potatoes, but in a production system with many processes, the gap widens as the CPU has to context switch.

Fireball3 · March 17, 2014

Thanks for taking your time to write those examples.

Finally found some time to work through them.

1.

Can we say that there is no possibility to call a function inside an IF without forking?

The function has to run prior to the IF and the IF itself is only comparing return values or variables.

This is probably the same for while-loops?

What would be a workaround to avoid this fork?

while [[ $(hour_match) -eq 1 ]] #
            do

Probably starting with "while [1 ]"

and calling the hour_match after the first loop-run?

2.

    check_LockFile
    hour_match
    if [[ $? -eq 0 && $LockFile -eq 0 ]]

When doing it like this:

"$? -eq 0" is working only if the function was called in the prior line?

So comparing multiple return codes won't be possible? The second value has to be a variable?

3.

Edit:

I noticed you use the variables like:

if [[ ${HOUR_MATCHED_RC} -eq 0 && $lockfile -eq 0 ]]

while in the whole script the variables are within quotes.

    if [[ "$hourMatch" -eq 0 && "$LockFile" -eq 0 ]]

Quotes = string, no quotes = integer?

If this is true, then I will have to stick either to the string-writing or to the integer-writing.

Then I would have to do:

if [[ "$?" -eq 0 && "$LockFile" -eq 0 ]]

or

if [[ $? -eq 0 && $LockFile -eq 0 ]]

Right?

4.

Conclusion:

While working with return codes seems pretty slight/slender it is not very transparent to the noob user.

Working with variables seems better especially if someone else is supposed to decipher it.

WeeboTech · March 17, 2014

This only really matters if you iterate over the same code many times in a loop.

Hence the reason why it's easy to get away with. i.e. until a system starts to bog and you see why.

$(function or command) causes a fork(split) no matter what.

calling a shell function and storing $? (return code) or storing a testable value in global variable is the only way to avoid the split.

As I mention, this matters most when you are calling something many times in a loop.

In the prior code I examined, there were 3 splits, to get the value of a string match.

Here you can go with what works best. I.E. it's either best for the human (understand-ability) or the computer (efficiency).

if [[ $? -eq 0 && "$LockFile" -eq 0 ]]

Is correct syntax. The quotes are to prevent an empty variable from causing a script to fail.

$? never needs to be quoted.

I use braces to be explicitly clear.

Especially since I do this all day long 7 days a week.

Fireball3 · March 17, 2014

OK, I think I can test the modifications now.

What is the best way to go for testing.

I read something about the debug function.

bash -x auto_s3_sleep.sh

I have to test that.

I would probably need to cycle through each variable...

WeeboTech · March 17, 2014

OK, I think I can test the modifications now.

What is the best way to go for testing.

I read something about the debug function.
bash -x auto_s3_sleep.sh
I have to test that.

I would probably need to cycle through each variable...

That's one way to do it.

With the line as follows you can do it on demand

[ ${DEBUG:=0} -gt 0 ] && set -x -v

you can enable it with

export DEBUG=1

./auto_s3_sleep.sh

or

DEBUG=1 ./auto_s3_sleep.sh

Fireball3 · March 17, 2014

You mentioned that already.

Obviously I didn't fully understand it.

...going back and looking up "export"...OK

Globally set DEBUG=1

I suppose this debug is different to the script $debug that is used for logging options.

The difference is only the way the script is called, isn't it?

sh -x -v auto_s3_sleep.sh

vs.

DEBUG=1 ./auto_s3_sleep.sh

Where is the benefit?

WeeboTech · March 17, 2014

You mentioned that already.

Obviously I didn't fully understand it.

...going back and looking up "export"...OK

Globally set DEBUG=1

I suppose this debug is different to the script $debug that is used for logging options.

The difference is only the way the script is called, isn't it?
sh -x -v auto_s3_sleep.sh
vs.
DEBUG=1 ./auto_s3_sleep.sh
Where is the benefit?

I have not looked at how $debug was used in the script

You can have different levels of debugging.

DEBUG=3 could mean something to sub shells or functions. Yes I've used it in that way.

I don't have examples. Certain loops and shell functions constructs do not show up fully in trace mode.

so I would put another one liner in the shell function and set the number higher in that function.

Then by doing DEBUG=2 or 3 they would show up.

Here's an interesting snippet to review.

Again, remember I always set ${P} to the script basename.


if [ ${DEBUG:=0} -gt 0 ]
   then set -x -v
        exec 9>/tmp/${P}.trace.$$
        export BASH_XTRACEFD=9
        export PS4='+ (${BASH_SOURCE}{LINENO}{FUNCNAME[0]-} '
fi

Fireball3 · March 17, 2014

Puh, that's really hardcore...

I have to google everything and puzzle something together.

And use the PS4 prompt to give more useful information like: export PS4='+(${BASH_SOURCE}{LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'

http://wiki.bash-hackers.org/scripting/debuggingtips

...munching...

Need help with this for loop

Recommended Posts

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Join the conversation