[MERGE] SMP implementation, part 1

Hello,

    Attached is the first part of the project making Squid SMP-scalable.
The implementation follows SMP design discussed a few months ago (quoted
below). The patch contains most of the promised Phase1 features. There
is more code brewing in the lab, but the attached code is ready to be
synced and merged with trunk. With these changes, you can run Squid in
SMP mode, utilizing many CPU cores, provided you do not mind cache
manager and caching inconsistencies.

A brief list of changes and to-dos is provided below. Besides the
acceptance review, I need help with these two questions:

1. Should we rename the squid.conf option specifying the number of
processes from "main_processes" to "worker_processes" or even "workers"?
The forked processes do almost everything the old daemon process did,
but that will change as we add more specialized processes. There is also
a special "Coordinator" process that is launched automatically in SMP
mode. It does not handle regular HTTP transactions.

2. We had to clone comm_openex into comm_open_uds because Unix Domain
Sockets do not use IP addresses. They use file names. We can unify and
simplify related code if we add the file name (struct sockaddr_un) to
Ip::Address. IIRC, Amos and I discussed this on IRC and decided that it
is OK to add sockaddr_un to Ip::Address (because it is used as a general
"network address" anyway), but I would prefer to hear more opinions
before altering Ip::Address.

Change log:

* Added main_processes squid.conf option to specify how many worker
processes to fork and maintain. Zero means old no-deamon mode. One means
the old non-SMP mode.

* Added support for process_name and process_number macros and
if-statement conditionals in squid.conf. Search for .pre changes for
documented details. These features allow the admin to configure each
worker process differently if needed.

* Support multiple workers listening on the same HTTP[S] port (port
sharing). This allows multiple workers to split the load without any
special rules.

* Support or prohibit port sharing for WCCP, DNS, ICP, HTCP, SMP, and
Ident protocols, depending on protocol-specific restrictions. Sharing is
implemented by registering listening socket descriptors with the
Coordinator process and obtaining them from the Coordinator as needed.
Here are protocol-specific notes:

   WCCP: Restricted to the Coordinator process due to how WCCP works.
   Workers do not need access to the WCCP code.

   DNS: Done by each worker with no sharing. Fixed source ports not
   supported unless each worker is given its own outgoing address
   because we do not want to match outgoing queries and incoming
   responses across processes.

   SNMP: Workers share incoming and outgoing sockets.

   ICP and HTCP _clients_: Cannot be supported in SMP environment
   unless each process has its own address (i.e., unique IP address
   and/or unique [ICP] port) because we do not want to match outgoing
   queries and incoming responses across processes.

   ICP and HTCP _servers_: share listening sockets.

   Ident clients do not need to share sockets because they use
   unique ports.

* Support management signals (squid -k ...) in SMP mode, acting as a
single Squid instance.

* Refork dying workers, similar to how we reforked dying process in
non-SMP daemon mode.

Detailed change descriptions are at
https://code.launchpad.net/~rousskov/squid/smp

To do (all these projects are in-progress now):

- SMP-aware cache manager.
- SMP-aware caching.
- Publish performance testing results.
- CPU affinity controls.

Thank you,

Alex.
P.S. The merge bundle is against trunk r10303. I will sync if approved.
The configure.in change is not meant for merging and will be excluded
when the code is synced and merged. I do not know how to exclude it in
the merge bundle.

On 02/11/2010 10:18 PM, Alex Rousskov wrote:
> Hello,
>
> I would like to implement the following changes to add initial SMP
> support to Squid. The changes are relatively simple but should allow us
> to reap a lot of SMP benefits in many real deployments. I hope to submit
> the results for review and trunk inclusion in about 45 days. More
> sophisticated changes will follow, guided by real and lab performance
data.
>
> The plan is based on the SMP-related discussions we had in the past few
> months. I factored in the apparent lack of developers available to
> tackle more ambitious designs, the current state of code, and our
> historical inability to handle huge projects without slipping off
> schedule and creating various disasters along the way.
>
>
> 1. Processes versus threads:
>
> The initial design is process- and not thread-based. Processes may be
> converted to threads and/or threads may be added to certain processes
> later. If we start with threads now, we may drown in problems caused by
> poor encapsulation of some of the major code pieces and
> thread-unreadiness of basic Squid libraries.
>
>
> 2. Building blocks:
>
> A Squid process dedicated to a subset of squid.conf options is a
> building block. The user can configure Squid to launch multiple such
> processes. Option subsets are likely to overlap a lot because many
> options would be the same for each process. In the initial
> implementation, _all_ subsets will be identical. In other words, all
> Squid processes will be identically configured and, hence, "do the same
> thing".
>
> The main Squid process will need to open http_port(s) and other
> listening sockets. It will either do it before forking child Squids or
> will pass open socket descriptors to child Squids via sendmsg(2).
>
> In the initial implementation, the admin will be able to specify which
> CPU core(s) should be used for Squid. Eventually, it would be possible
> to map individual building blocks to individual CPU cores.
>
>
> 3. What is expected to work:
>
> SMP-scalable performance on general workloads. For example, if you have
> 8 CPU cores, you can utilize all of them.
>
> Squid will behave as a single instance with respect to misses,
> reconfiguration, access logging, and mgr:info part of the cache manager
> interface.
>
>
> 4. Limitations:
>
> Caching in the initial implementation is not shared and not synchronized.
>
> Options that require exclusive, single-process access such as a single
> source port for HTCP queries will not be supported in SMP mode.
> Eventually, the associated functionality can be adjusted to work with
> multiple processes or threads.
>
> When logging via a pipe to a program, multiple program instances would
> be launched.
>
> I am sure other limitations will surface.
>
>
> 5. Next steps:
>
> After the initial design is implemented, we will add support for shared
> or at least synchronized cache. Also, more cache manager pages will
> aggregate information from all processes.
>
>
> Please review.
>
> Thank you,
>
> Alex.
>
>

On 02/20/2010 07:14 PM, Alex Rousskov wrote:
> Hello,
>
> If you recall, I am working on Squid that starts multiple processes,
> each doing similar things. Even with this simple design, folks want to
>
> (a) have differently configured processes (e.g., a process that is
> dedicated to a given http_port or even a cache_dir option); and
>
> (b) bind processes to specific CPU cores (i.e., support CPU affinity)
>
>
> I propose the following configuration approach that I think is simple to
> implement but allows a lot of flexibility:
>
> 1. Each forked process gets a unique process name, which is just a
> number from 1 to N. The process knows its name. If a forked process dies
> and is reforked, the reforked process keeps the original name.
>
> 2. The squid.conf parser substitutes ${process_name} strings with the
> process name doing the parsing. This substitution is performed before
> individual options are parsed.
>
> 3. The squid.conf parser supports if-statement blocks. Each if-statement
> must start on its own line (as if there is an option called "if"). Each
> if-statement block ends with "endif" on its own line (as if there is an
> "endif" option). The only two supported conditions for now are a simple
> comparison:
>
> if ${process_name} = 1
> ... regular squid.conf options for the first forked process ....
> ... regular squid.conf options for the first forked process ....
> ... regular squid.conf options for the first forked process ....
> endif
>
> and a set membership test, for when we want to specify options for
> multiple processes:
>
> if ${process_name} in {1,7,8}
> ... regular squid.conf options for the selected forked process ....
> ... regular squid.conf options for the selected forked process ....
> ... regular squid.conf options for the selected forked process ....
> endif
>
> If the condition is false, the parser skips all regular squid.conf
> options inside the block until the matching endif. Otherwise, the parser
> behaves as if the if-statement was not there.
>
> This approach supports process-specific options without rewriting the
> existing options or the squid.conf parser. I think the implementation is
> straightforward, even if we want to support nested if-statements. We
> just push the current if-statement condition on stack and either skip or
> honor options until we find endif and pop the current condition.
>
> As a side effect, we can use the same if-statement approach to quickly
> disable large portions of the configuration file using conditions that
> are always false.
>
>
> 4. CPU affinity is supported using a new cpu_affinity option that
> specifies either a single CPU core ID (1..C) or the affinity mask:
>
> # start this (and each) process on its own core:
> cpu_affinity core=${process_name}
>
> # use any first four cores for this (and each) process:
> cpu_affinity mask=0xF
>
> # place process5 on CPU core1:
> if ${process_name} = 5
> cpu_affinity core=1
> endif
>
>
> Any objections, improvement suggestions, or better ideas?
>
> Thank you,
>
> Alex.
>