Re: [RFC] Squid process model

On 11/01/2013 07:11 AM, Amos Jeffries wrote:

> As some of you are no doubt aware that one of the design issues we are
> facing with Squid these days is the process model. The current model has
> a very init.d centric design and shoots itself in the foot when
> encoutering third-party daemon management systems such as upstart,
> systemd, and a few other less popular ones. Not that it supports many
> uses of init.d very well either.

I agree that we can and should do better.

> I have been thinking of updating the -N, -n -z, -k command line options
> behaviour very slightly to make things a bit more flexible in a backward
> compatible way.
>
>
> -N - no-daemon mode
>
> Today -N takes no parameters and disables all forking. This is great for
> debugging a worker, but not much else.
>
> If we add an optional parameter that takes the process kind name
> (worker, coordinator, disker, etc.) and starts a process with that type

Manual "kind-specific" process starting (and running) is way too
complicated to implement correctly and not useful enough to support IMO.
Most SMP Squid processes will not work today in isolation, and I can
think of very few use cases where starting them manually is useful in
the real world. On the other hand, I am sure we will get lots of
complaints and bug reports that something does not work when folks try
to run a single process [kind] that cannot run in isolation.

It would be a strategic error to expose Squid guts like that IMO. We
should do our best to present Squid instance as a monolithic proxy
object, with a single configuration file, regardless of what we do
internally to implement that proxy. We should not encourage folks to
fiddle with implementation details such as various obscure process roles.

> this will allow us to debug individual process types and allow
> third-party daemon manager to kick off specific processes by type (if
> they know how).

One can debug individual process types now with debug_options and
squid.conf conditionals. And the real problem here is not _starting_ a
specific process, but making that specific process _work_ when the stuff
it needs from other processes is not available.

Moreover, technically, one can probably start individual Squid processes
today already! They just need to create "(squid-N)" links to the Squid
executable and start Squid using those links. No new options are needed.
Not that we should support or even document that for the reasons
discussed above, but it may already be possible. I have not tested that
so I am not sure whether the initial Squid process does something
important that will be missing when starting "(squid-N)" programs from
the command line.

> To integrate with upstart/systemd cleanly we would need a mode which
> starts a process instance without forking the main processinto the
> background.

Yes, this would make clean integration easier. The following three
approaches seem reasonable to me:

a) Add an optional parameter to -N to be able to switch between
   "no forking" and "no daemon" semantics. For example, if "-N" is
   interpreted as "No", we can have:
   -N # default (to be decided)
   -N daemon # Master process runs in foreground but SMP allowed
   -N fork # No forking at all and, hence, no SMP.

b) Change -N semantics from current "no forking" to "no daemon"
   and add another option to enable "no forking" semantics.

c) Leave -N semantics as current "no forking"
   and add another option to enable "no daemon" semantics.

I am not sure which approach is the best.

> -n - Windows service name
>
> The Windows build of Squid requires a -n option to point at the
> particular named service which is running in the background. Which
> defaults to the name "squid" when omitted.
>
> Making this option available outside Windows shows some promise. With
> the service name being used as prefix for shm_*() paths, default pid
> file name and similar things which are required to be identical between
> all processes in a Squid instance this will restore the ability to run
> multiple independent Squid services on the same machine regardless of
> whether SMP support is used or not.

Sounds good to me.

> -z - store initialization mode
>
> The -z execution is currently able to interfere with other executions
> and instances of Squid set running at the same time or shortly after it
> exits while the sub-processes it forks off are still running.

Yes.

> Due to what it is currently scoped to do (simply and specifically to
> initialize and repair structural damage to a cache_dir)

Yes.

> there is no
> reason for -z to interact with any other Squid instance or even to block
> on a previously existing instance already running.

I disagree. It would be conceptually wrong and, in some cases,
technically impossible, to "initialize and repair structural damage to a
cache_dir" that is being used by another Squid instance.

> We could make two changes here to improve the situation:
> 1) make it support the -n mode change above, with a different default
> service name "squidZ" to distinguish from other non-z instances.
> Allowing an overlap of execution with an already running or later
> started process.

To protect its integrity, a cache_dir being used by one instance should
not be used by anybody else. Breaking that exclusive lock is the wrong
model, even if we add options to make it technically possible.

> 2) make the main executed process default to either having -N properties
> when SMP is disabled, or to being the coordinator when SMP is enabled.
> The key criteria here is that the foreground process should never exit
> until the -z processing by each forked sub-process is completed.
> Preventing existing init scripts which execute -z sequentially from
> breaking the main instance being started shortly after.

We have discussed that before: No matter what we do here, many (most?)
old startup scripts will need to be adjusted anyway because they do not
correctly detect Rock cache_dir presence. Thus, I think "squid -z"
should use the same default as "squid" when it comes to "daemon mode",
for consistency sake.

However, if others think that the default should be changed again, I
will not object to that change.

I would also like to propose an alternative:

Make "squid -z" a no-op (Squid would just exit immediately after
detecting that option). Teach Squid to always prime cache_dirs at
startup. This will remove all problems discussed above. Moreover, it
will make admins live easier because they do not need to think about
writing correct scripts to detect cache_dir validity. It was probably a
mistake to add "squid -z" in the first place. We can correct it now.

The performance cost for always checking cache_dirs at startup is
negligible for Rock.

If that cost is too much for ufs-based dirs, we can alter their
algorithm to do nothing if top-level directories exist. This will be
very similar to what the vast majority of startup scripts are doing now
(because those script do not verify that _all_ UFS cache_dir subdirs are
present when deciding whether to run -z). Alternatively, we can teach
UFS-based dirs to create cache_dir sub-directories runtime if they are
needed but are missing (triggered by UFS file creation errors). Both
approaches would reduce performance overhead to negligible levels.

If Squid needs additional information to create missing cache_dirs
correctly, we can add more options to squid.conf to supply that information.

> -k - signalling a running instance
>
> At present this is just a command line tool to generate signals for a
> running instance. If the above changes are made it will require a small
> behavioural change to support the -n option and operate for all OS as it
> does in Windows. By taking a targeted instance/service name and sending
> the signal specifically to the main signal handling process of that
> instance.

Sounds good.

Thank you,

Alex.
Received on Fri Nov 01 2013 - 16:09:43 MDT