*** 24.12.19 bugs

- Fixed a bug that would cause the *condor_startd* to crash if a custom :macro:`DOCKER`
  docker wrapper emitted unexpected output to the "images" sub-command.
  :jira:`3596`

- Fixed a bug where :tool:`condor_status` would not adjust the Name column width to fit data
  for some output formats. This bug would result in columns after the Name column not
  being properly aligned when the longest name exceeded 34 characters.
  :jira:`3255`

- Fixed a bug where :tool:`condor_vacate` would fail to find the address of the *condor_startd*
  when provided a slot name. With this change the *condor_collector* now searches through both
  Startd Daemon Ads and Startd Slot Ads when handling a locate query.
  :jira:`3580`

- Fixed a bug in :tool:`condor_q` match analysis that would always count sub-expressions that
  evaluated to a double or integer as not matching.
  :jira:`3601`

*** 24.0.19 features

- HTCondor tarballs now contain `Pelican 7.24.0 <https://pelicanplatform.org/releases>`_

- The condor package now requires pelican-7.24.0.

*** 24.0.19 bugs

- Fixed a bug where an NVIDIA GPU that has been subdivided in MIG sub-devices
  would not properly pass the device names in the various NVIDIA related
  environment variables.
  :jira:`3567`

- Fixed a performance problem with the classad2 python bindings that
  caused the parseAds methods to run in n-squared time.
  :jira:`3632`

- Fixed a bug where :subcom:`allowed_execute_duration` would possibly not be
  respected for jobs that did not do input file transfer.
  :jira:`3633`

- Fixed a bug for **batch** grid universe jobs where submitting to a remote
  system would fail if the username on the remote system contained a dot.
  :jira:`3581`

- Fixed a bug where some of the values reported for job execution time were
  enormous if the Access Point daemons restarted while the job was running.
  :jira:`3592`

- Fixed a bug where an administrator couldn't query a daemon's configuration
  using :tool:`condor_config_val` when authentication is required for READ
  level authorization.
  :jira:`3572`

- The :meth:`htcondor2.send_command` method no longer incorrectly requires
  a ``target`` parameter.
  :jira:`3597`

*** 24.12.18 bugs

- Improve AMD GPU detection when using ROCM6/HIP libraries and fix a potential crash
  when using the HIP libraries with the default detection options passed by the *condor_startd*.
  :jira:`3533`

- Fixed issue where jobs could match Execution Points utilizing :macro:`STARTD_ENFORCE_DISK_LIMITS`
  only for them to immediately be kicked off due to unavailable disk space due to quantization
  issues between HTCondor and LVM. This would could cause lots of unnecessary job churn when
  Execution Points are very busy.
  :jira:`3490`

- Fixed a bug where an job with a circular :ad-attr:`Requirements[type=Job]` expression
  would result in a crash in :tool:`condor_q` when the ``-better`` option was passed.
  The crash was fixed, and :tool:`condor_submit` was changed to report an error
  when the submit file has a :subcom:`requirements` command that
  references :ad-attr:`Requirements[type=Job]`.
  :jira:`2629`

- Fixed a bug in *condor_status* where the Offline-GPUs column of the ``-gpus``
  output was always empty.
  :jira:`3558`

- Fixed a bug in :meth:`htcondor2.Schedd.history` which prevented its `since`
  argument from being the string representation of an expression.
  :jira:`3435`

- Fixed a bug where a backfill p-slot would have no assigned GPUs after
  a reconfig.
  :jira:`3072`

*** 24.0.18 features

- Enable the in-memory cache option in newer versions of the scitokens-cpp
  library. This allows the library to continue operating when the on-disk
  cache is not useable.
  :jira:`3560`

- HTCondor tarballs now contain `Pelican 7.23.0 <https://pelicanplatform.org/releases>`_

- The condor package now requires pelican-7.23.0.

*** 24.0.18 bugs

- Fixed a bug in :tool:`condor_submit` that could expand the :subcom:`executable`
  path incorrectly when using late materialization and a different executable
  for each job.
  :jira:`3546`

- Fixed a bug where new STARTD_CRON values didn't trigger an immediate
  update of slot ads to the collector when configured to do so.
  :jira:`3526`

- Update Annex's knowledge about the Delta HPC system.
  :jira:`3559`

*** 24.12.17 bugs

*** 24.0.17 bugs

- Fixed :meth:`htcondor2.Schedd.refreshGSIProxy` to use correct
  variables and not immediately fail.
  :jira:`3503`

- Fixed bugs in :tool:`condor_history` that made it very slow on
  filesystems with rate-limits on I/O operations.
  :jira:`3512`

*** 24.12.16 bugs

- Fixed a bug where specifying a scope or audience when storing a
  Vault-managed credential results in a stored token that can't be used.
  :jira:`3506`

- In the *condor_schedd* when a transform attempts to set an immutable attribute
  more than once, the second attempt be quietly ignored rather than failing job materialization.
  :jira:`3495`

- Fixed a bug where a backfill p-slot could get into a state where it
  would advertise that it had capacity, but refuse every claim with
  a log message indicating that it had zero CPUs.
  :jira:`3072`

- Fixed a bug in the shared-port daemon that caused new network connections
  to fail if only a few bytes are immediately available to read.
  :jira:`3420`

*** 24.0.16 features

- HTCondor tarballs now contain `Pelican 7.22.0 <https://pelicanplatform.org/releases>`_

- The condor package now requires pelican-7.22.0.

- HTCondor tarballs now contain `Apptainer 1.4.5 <https://github.com/apptainer/apptainer/releases/tag/v1.4.5>`_

- The condor RPM package now requires at least apptainer version 1.4.5.

*** 24.0.16 bugs

- Fixed a bug where a job with a floating point value for :ad-attr:`RequestMemory`
  or :ad-attr:`RequestDisk` might not match the slot created to run it.
  :jira:`3423`

- Fixed a bug that would cause a crash in the *condor_rooster* and *condor_defrag* when many slots
  would have identical :macro:`ROOSTER_UNHIBERNATE_RANK` expressions, or if was
  not constant (e.g. used random() or time()).
  :jira:`3436`

- Removed memory leak from :meth:`htcondor2.JobEventLog.events`.
  :jira:`3474`

- Changed :tool:`condor_history` so that when it prints jobs in ``-long`` format
  it prints them with attributes sorted alphabetically like :tool:`condor_q` does.
  :jira:`3481`

- Fix Logical Volume setup timeout on Execution Points enforcing disk usage
  for jobs that have a large :subcom:`request_disk` value (``800+ GB``).
  :jira:`3432`

- Fixed issue with Execution Points enforcing disk usage generating an
  excessive amount of metadata archives filling up ``/etc/lvm/archive``.
  :jira:`3488`

- Added :macro:`FILETRANSFER_PLUGIN_CLASSAD_TIMEOUT` for administrators
  to set a longer timeout for plugins to detect their health.
  :jira:`3455`

- Fixed a bug where docker universe jobs would bring back a
  file named ``.docker_stderror`` back to the AP after job completion.
  :jira:`3424`

- Fixed a bug where the values of the attributes :ad-attr:`LocalJobsIdle` and
  :ad-attr:`LocalJobsRunning` in the submitter ad were reversed.
  :jira:`3456`

*** 24.12.15 features

- :tool:`condor_submit` will now report an error when :subcom:`output_destination` is
  not a URL and suggest an alternative.
  :jira:`3385`

*** 24.12.15 bugs

- Improve compatibility with versions of HTCondor that attempt to set
  attribute :ad-attr:`OsUser` in the job ad.
  :jira:`3415`

- Calling :meth:`htcondor2.Schedd.submit` with a list of dictionaries with dissimilar
  keys will no longer mangle the values (and instead ignore the extra keys,
  as implied by the documentation).
  :jira:`3351`

*** 24.0.15 features

- HTCondor tarballs now contain `Pelican 7.21.1 <https://pelicanplatform.org/releases>`_

- The condor package now requires pelican-7.21.1.

- HTCondor tarballs now contain `Apptainer 1.4.4 <https://github.com/apptainer/apptainer/releases/tag/v1.4.4>`_

- The condor RPM package now requires at least apptainer version 1.4.4.

*** 24.0.15 bugs

- Fixed a bug that could cause the AP to fail to read job credential files.
  :jira:`3377`

- Fixed bugs that could cause a crash in the authentication code.
  :jira:`3394`

- Fixed a bug where the *condor_negotiator* would fail to contact the
  *condor_schedd* to perform matchmaking if its :macro:`ALLOW_CLIENT`
  configuration parameter didn't authorize the identity
  ``submit-side@matchsession``.
  :jira:`3378`

- Fixed a bug where tools like :tool:`condor_status` would print an incorrect
  value for the ``-af:r`` option when the value to be printed was a single
  attribute reference.
  :jira:`3347`

*** 24.12.14 features

- Added a new EP configuration knob, :macro:`DOCKER_TRUST_LOCAL_IMAGES`
  which defaults to false.  Setting this to true allows users to run
  docker images which have been pre-staged in the EPs docker cache even
  if the image does not exist in a repository, or if the user does not
  have permission to pull from that repository.
  :jira:`3315`

- The *condor_schedd* will now include the address of a *condor_credd*
  that is running under the same *condor_master* in its ClassAd and
  address file. This allows the submission process to get the address
  of the *condor_credd* from the *condor_schedd* for some situations
  where credentials must be stored as part of job submission.  The
  Kerberos local issuer will now use this mechanism and no longer query
  the collector for the address of the *condor_credd*.
  :jira:`3281`

*** 24.12.14 bugs

- Fix interoperability problem between HTCondor-CE 24 and 25 which
  manifests as a Job Router crash when upgrading the CE to HTCondor 25
  :jira:`3355`

- Fixed a bug in :meth:`htcondor2.Schedd.submit` where single-entry itemdata could be
  truncated to its first character.
  :jira:`3272`

- Fixed a bug in :meth:`htcondor2.Schedd.submit` where the ``$(step)`` submit variable
  wasn't being set.
  :jira:`3272`

- Fixed a bug in :meth:`htcondor2.Submit.itemdata` causing multi-entry itemdata to be
  returned incorrectly.
  :jira:`3272`

- Fixed a bug where using :subcom:`max_idle`, :subcom:`container_image` 
  and :subcom:`transfer_input_files` could result in the container only
  being transferred along with the first job.
  :jira:`3092`

- Changed the documentation for :meth:`classad2.ClassAd.matches` and
  :class:`classad2.ClassAd` to match their implementations; the original
  documentation was in error, and both are actually backwards-compatible
  with the first version of the Python bindings.  Made other minor changes;
  see the ticket for details.
  :jira:`3328`

- Calling ``classad2.ExprTree(None)`` no longer results in an invalid
  :class:`classad2.ExprTree`, preventing segmentation faults when using the object.
  :jira:`3319`

- Fixed a bug where :tool:`condor_history` could take many minutes to read a single line
  from the history file when the line is many megabytes long. This was causing
  ingestion of epoch ads into a database to timeout.
  :jira:`3299`

- Fixed a bug where if input file transfer failed, occasionally no
  descriptive error message about the failure would make it back
  into the job hold reason.
  :jira:`3327`

- Fixed a very longstanding bug where extremely fast machines would
  overflow a 32 bit counter and return -1 for the :ad-attr:`KFlops`
  slot attribute.
  :jira:`3288`

- The ``gpu`` and ``gpu-debug`` queues at Anvil are no longer represented in
  :tool:`htcondor annex` as whole-node queues, fixing a problem where the
  "whole node" would have 4 GPUs and only 1 CPU.
  :jira:`3324`

*** 24.0.14 bugs

- Fix problem running PyTorch jobs on multiple GPUs with
  newer versions of the CUDA library by providing long GPU IDs
  in the ``CUDA_VISIBLE_DEVICES`` environment variable
  :jira:`3350`

- Calling :meth:`htcondor2.Schedd.history` on an object whose corresponding
  daemon can't be contacted will no longer cause a segmentation fault.
  :jira:`3314`

- Executions points using disk enforcement failed when provided backing
  :macro:`LVM_VOLUME_GROUP_NAME` and :macro:`LVM_THINPOOL_NAME` contained
  hyphens.
  :jira:`3334`

- Add a timeout for all HTTP operations in the grid universe (affects
  **arc**, **ec2**, and **gce** grid types).
  :jira:`3300`

- Fixed a bug where administrative tools (e.g. :tool:`condor_drain`) can fail if
  authentication is not required for queries to the *condor_collector*.
  :jira:`3301`

- The *condor_startd* now correctly removes docker containers that have escaped from
  docker universe jobs launched by *condor_starter*\'s that have exited uncleanly.
  :jira:`3338`

- Annexes no longer download a default configuration tarball, making them
  more robust.  This required upgrading the default version of HTCondor run
  by annexes to 25.1.0, which can figure the details out on its own.  On
  Delta, annexes now give Delta additional time to start file-transfer plug-ins,
  so they should be more-reliably available there.
  :jira:`3220`

- :tool:`condor_adstash` no longer skips entire history files when it
  encounters a line that cannot be decoded to UTF-8.
  :jira:`3335`

- Fixed a bug in *blahpd* when ``/tmp`` is mounted with the ``noexec`` option.
  :jira:`3343`

*** 24.12.13 bugs

- Fixed bug where the message ``Processing new events...`` would briefly
  flash while running :tool:`condor_watch_q`
  :jira:`3244`

*** 24.0.13 features

- HTCondor tarballs now contain `Pelican 7.20.2 <https://pelicanplatform.org/releases>`_

- The condor package now requires pelican-7.20.2.

*** 24.0.13 bugs

- Fixed a bug where space and comma would be included in the list of
  separators for itemdata even if the itemdata had been supplied with
  the ASCII "unit separator".  This would cause itemdata entries containing
  spaces (or commas) to be incorrectly interpreted as multiple items, which
  could manifest as parse errors.  You can work around this bug if only one
  of your entries has spaces and/or commas by moving that entry to the end
  of the line (or dictionary, if you're submitting itemdata via Python).
  :jira:`3272`

- Fixed false positive reporting of ClassAd unit specifier test in
  :tool:`condor_upgrade_check`.
  :jira:`3276`

*** 24.12.4 features

- :meth:`htcondor2.Collector.locate` now checks the :attr:`Machine` attribute
  (instead of :attr:`Name`) when trying to find *condor_startd* daemons,
  if the ``name`` argument is not the name of a slot.  This allows programmers
  to call :meth:`htcondor2.Collector.locate` using the fully-qualified domain
  name and get a result from a collector whether or not
  :macro:`ENABLE_STARTD_DAEMON_AD` is set.
  :jira:`2911`

- Add new configuration knob, :macro:`LOCAL_UNIVERSE_CGROUP_ENFORCEMENT`, which
  defaults to false.  When true, and running on a cgroup enable system, local
  universe jobs must specify :subcom:`request_memory`, and if the job exceeds that
  limit, it will be put on hold.
  :jira:`3170`

- :macro:`HISTORY_HELPER_MAX_HISTORY` default raised from 10,000 to 2,000,000,000
  effectively removing the limit.  We now believe that running large
  remote history queries will not have a negative impact on the *condor_schedd*
  :jira:`3215`

- :tool:`condor_userprio` now no longer needs the negotiator to be running
  if it only needs the collector to get userprio data.
  :jira:`3221`

- Docker universe jobs now support the config parameter :macro:`DOCKER_NETWORK_NAME`
  which defaults to "docker0".  Setting this appropriately allows :tool:`condor_chirp` 
  and :obj:`htcondor.htchirp` to work correctly inside a docker universe job.
  :jira:`3197`

- :tool:`condor_watch_q` will now exit upon key press from keyboard.
  :jira:`3199`

- Added new :ad-attr:`NumInputTransferStarts` and :ad-attr:`NumOutputTransferStarts`
  counters to the job record.
  :jira:`3194`

- If the file listed in :subcom:`log` is in a directory that does not exist, HTCondor
  will now try to create that directory (as the user).  Previously, it ran the
  job without a user log file.
  :jira:`3210`

- Added new knob, :macro:`STARTER_NESTED_SCRATCH`, which defaults to false.  When false,
  the job scratch directory (named dir_XXX) is the immediate child of the :macro:`EXECUTE`
  directory.  When true, the job's scratch directory is a subdirectory thereof, named
  "scratch", with the intention that all condor metadata files will be moved to peer 
  directories, and not pollute the job's scratch directory with condor control files.
  :jira:`3080`

- Added optional second argument for the macro expansion function ``$BASENAME()`` that
  specifies the file extension or suffix to remove.
  :jira:`3206`

- :tool:`condor_adstash` now parses and stores per-attempt transfer error data when
  reading transfer epoch history.
  :jira:`3122`

- On Windows, :ad-attr:`ResidentSetSize` and thus :ad-attr:`MemoryUsage` will now
  be updated in the startd as frequently as :ad-attr:`ImageSize` so that the default
  :macro:`PREEMPT` and :macro:`WANT_HOLD` policy expressions will evict jobs
  that go over :ad-attr:`Memory` in a timely manner.
  :jira:`3224`

*** 24.12.4 bugs

- Allow :tool:`condor_submit` -i to work with :subcom:`shell`.
  :jira:`3208`

- Fixed bug when :macro:`STARTER_NESTED_SCRATCH` is true and
  :macro:`SINGULARITY_TARGET_DIR` was also set.
  :jira:`3243`

- Fixed bug that prevented emails being sent to users when jobs went
  on hold and :subcom:`notification` was set to error.
  :jira:`3219`

- Fixed bug introduced in 24.11 that caused the *condor_schedd* to
  occasionally crash when DAGMan completed.
  :jira:`3250`

*** 24.0.12 features

- Updated :tool:`condor_upgrade_check` to test for well known gotchas
  between v24 and v25 of HTCondor installations.
  :jira:`3209`

*** 24.0.12 bugs

- Fixed a bug where the *blahpd* would fail when using Python 3.12 or later.
  :jira:`3225`

- Fixed bug where jobs held due to exceeding disk usage had the same
  :ad-attr:`HoldReasonCode` and :ad-attr:`HoldReasonSubCode` as jobs
  that exceeded memory usage.
  :jira:`3248`

*** 24.11.2 features

- Added new job ClassAd attributes :ad-attr:`NumVacates` and :ad-attr:`NumVacatesByReason`.
  These attributes provide counts about why a job left the running state without
  completing (i.e. was vacated from the execution point).
  :jira:`3204`

- Added new "notification = start" option to *condor_submit*, which
  sends an email when the job starts for the first time.
  :jira:`3133`

- New configuration parameter :macro:`TRUSTED_VAULT_HOSTS` can be used to
  restrict which Vault servers the *condor_credd* will accept credentials for.
  :jira:`3136`

- The new Python API now includes `htcondor2.ping()`, which operates like
  `htcondor.SecMan.ping()` in the old API.
  :jira:`3180`

- The :tool:`htcondor annex` tool now has (limited) support for AWS' EC2 annexes.
  The ``condor_annex`` tool has been withdrawn; use :tool:`htcondor annex`
  instead.
  :jira:`1630`

- Added new python method "get_claims" to the schedd object, which returns
  the classads of the claimed slots.
  :jira:`3181`

- Add --version flag to the :tool:`htcondor` tool.
  :jira:`3091`

*** 24.11.2 bugs

- When responding to a ping request for the ALLOW authorization level,
  daemons no longer require authentication.
  :jira:`3195`

*** 24.0.11 features

- Initial support for Debian 13 (trixie).
  :jira:`3212`

*** 24.0.11 bugs

*** 24.10.3 bugs

- Fixed a bug introduced in version 24.10.2 that caused
  `condor_store_cred add` to fail.
  :jira:`3213`

*** 24.10.2 features

- In the *condor_job_router*, the old ClassAd-based route syntax
  (specified using ``JOB_ROUTER_ENTRIES`` and ``JOB_ROUTER_DEFAULTS``)
  is no longer supported.
  :jira:`3118`

- Added new :tool:`condor_dag_checker` tool for users to check DAG files
  for syntactical and logical errors prior to submission.
  :jira:`3088`

- Improvements to :tool:`condor_q` for held jobs. The hold code and subcode are
  now displayed as part of the ``-hold`` option. A new option ``-hold-codes``
  displays the first job for each unique hold code and subcode.
  :jira:`3127`

- Added new ``-lvm`` option to :tool:`condor_status` to view current disk usage
  of slots enforcing disk limits. This option can be paired with ``-startd`` to
  show information about execution points enforcing disk limits.
  :jira:`3119`

- The new Python API now includes :py:meth:`htcondor2.disable_debug`, which is
  intended interactive use (after debugging a problem).
  :jira:`3003`

- Some errors on the EP that occurred after the AP had released the
  corresponding claim could cause a slot to remain claimed until the job lease
  timeout had expired.  This change should reduce incidents of this behavior.
  :jira:`3028`

- Improvements to observability of common files transfer, including new entries
  in the shadow and starter daemon logs; a new CommonFiles event in the
  job/user event log; and a new transfer entry in the epoch history.
  :jira:`3052`

*** 24.10.2 bugs

- Fixed a bug in :tool:`condor_q` default output where counts of jobs could be truncated to 6 digits.
  :jira:`3106`

- Fixed a bug introduced in HTCondor version 24.8.0 where a job in Suspended
  status wouldn't change to Idle status when evicted from an EP.
  This resulted in the job not being considered for scheduling, among other
  problems.
  :jira:`3174`

- Execution Points enforcing disk limits will now subtract the size of
  pre-existing logical volumes from the advertised available disk. Any
  logical volumes associated with HTCondor are not subtracted.
  :jira:`3119`

- Fixed a bug where the *condor_credd* mistakenly thought a Vault-managed
  OAuth2 credential was a plain user-provided access token.
  :jira:`3084`

- Attempting to send common files to startds whose sinful string is more than
  256 characters will no longer cause a shadow exception.
  :jira:`3128`

- Fixed a memory leak in the *condor_schedd* that could be triggered by checkpointing.
  :jira:`3104`

- Fixed an issue where a job may take an additional 20 minutes to be scheduled to run after leaving cool-down mode.
  See configuration knob :macro:`SYSTEM_ON_VACATE_COOL_DOWN` for more information about job cool-down mode.
  :jira:`3059`

*** 24.0.10 bugs

*** 24.9.2 features

- Added new job ClassAd attribute :ad-attr:`TransferInputFileCounts`.
  :jira:`3024`

- Added new :macro:`SCHEDD_DAEMON_HISTORY` file for the Schedd to periodically write
  historical ClassAd records into. These records can be queried via :tool:`condor_history`
  using the new **-daemon** option or via :py:meth:`htcondor2.Schedd.daemonHistory`.
  :jira:`3061`

- :tool:`condor_watch_q` will now display tracking information for DAGMan jobs specified
  via the **-clusters** option.
  :jira:`3068`

- Improved logging on the EP when a slot cannot be claimed because the :ad-attr:`Start`
  expression evaluates to false.  When this happens, analysis of the slot :ad-attr:`Requirements[type=Machine]`
  expression will be written to the ``StartLog``.
  :jira:`3033`

*** 24.9.2 bugs

- Fixed a bug which could cause unnecessary activation failures if the previous
  job in the slot failed to transfer its output.  This would manifest as slots
  being in the claimed/idle state for far longer than necessary.
  :jira:`3073`

- Fixed a bug in the Vault credential monitor where access tokens were failing
  to be generated from Vault tokens when :macro:`AUTH_SSL_CLIENT_CAFILE` and/or
  :macro:`AUTH_SSL_CLIENT_CADIR` were undefined.
  :jira:`3086`

- Fixed a bug in :class:`htcondor2.Schedd` where it didn't work to use a
  ``job_spec`` parameter to specify a cluster ID as an integer, as a string
  without a proc ID, or in a list of such strings.
  :jira:`2979`

*** 24.0.9 features

- Initial Support for Enterprise Linux 10, including the x86_64_v2 platform.
  :jira:`3090`

*** 24.0.9 bugs

- The results of ``key in htcondor2.param`` and
  ``key in htcondor2.param.keys()`` now match for keys which are defined to
  have no value.  (Previously, such keys would be returned by ``keys()``.)
  :jira:`3085`

*** 24.8.1 features

- On Linux systems with cgroups enabled, jobs are now put in a ".scope"
  sub-cgroup of the per-job ".slice" cgroup.  This makes it easier for
  pilot or glidein systems to further subdivide the job's cgroup.
  :jira:`3008`

- On Linux systems, added support to put each condor daemon in its
  own cgroup with the knob :macro:`CGROUP_ALL_DAEMONS`
  :jira:`3032`

- The execute point now sets the execute permission bit on
  the :subcom:`executable` even when it was transferred by a plugin.
  This is helpful when using pelican or osdf to transfer the
  job's main executable.
  :jira:`3020`

- Add a new configuration knob, :macro:`STARTER_SETS_HOME_ENV` which defaults to 
  true.  When true, the job will have the **HOME** environment variable
  set to whatever it is on the system.  When false, HOME will not
  be set to anything.
  :jira:`3010`

- Added new ``halt`` and ``resume`` verbs to :tool:`htcondor dag` for
  first class way to halt a DAG.
  :jira:`2898`

- Added new :class:`htcondor2.DAGMan` class to the python API for sending
  commands to running a DAGMan process.
  :jira:`2898`

- Added :macro:`DAGMAN_NODE_JOB_FAILURE_TOLERANCE` to inform DAGMan when to
  consider a placed job list as failed when job failures occur.
  :jira:`3019`

- :tool:`htcondor ap status` will now show the :ad-attr:`RecentDaemonCoreDutyCycle` of
  each reported Access Point's *condor_schedd*.
  :jira:`3009`

- :tool:`condor_adstash` can now be configured to fetch a custom projection of attributes
  for job (epoch) ClassAds.
  :jira:`2680`

- :tool:`condor_status` will now accept ``-totals`` (previously just ``-total``) to better
  match other tools with the similar option.
  :jira:`3044`

- Improve diagnostics in the shadow when it fails to activate a claim.
  :jira:`3035`

- The directory for :macro:`LOCAL_UNIV_EXECUTE` is no longer made
  world-writable.
  :jira:`3036`

- Augment the `libvirt_simple_script.awk` script to provide needed
  UEFI boot information for ARM virtual machines.
  :jira:`3006`

- The :tool:`condor_upgrade_check` script has been folded into the main condor package.
  :jira:`2995`

*** 24.8.1 bugs

- Fixed a bug in the EP preventing a claimed slot from being re-used to run multiple jobs.
  The ability for an AP to run multiple jobs on the same claimed slot (i.e. without needing
  to go back to the central manager) is a critical scalability feature in HTCSS, especially
  when running large numbers of short-running jobs.  The bug fixed here was introduced in 
  HTCondor version 24.5.1, so if you are running HTCondor v24.5.x, v24.6.x, or v24.7.x,
  and run large numbers of short jobs, please consider upgrading.  See the JIRA ticket
  for additional workarounds if you cannot upgrade.
  :jira:`3045`

- On Linux and macOS, when using dataflow jobs, HTCondor now checks the modification
  times of dataflow nodes with sub-second accuracy.  Previously, it just
  used seconds, which means that it might incorrectly not skip a dataflow
  job that it should have skipped if the output file was written in the
  same second as the input file.
  :jira:`3027`

- Fixed :tool:`condor_watch_q` to output a useful error message and not
  exit when one of :subcom:`log` files associated with jobs being tracked
  does not exist.
  :jira:`2978`

- Removed job attribute ``ToE``.
  It has been replaced by job attributes :ad-attr:`VacateReason`,
  :ad-attr:`VacateReasonCode`, and :ad-attr:`VacateReasonSubCode`.
  :jira:`2974`

- The ``SlotName`` field in the job event log is now correct in the
  case where a *condor_startd* has a non-default name.
  :jira:`3047`

- Fixed a bug where :meth:`htcondor2.enable_debug()` would cause the
  Python interpreter to exit if the debug log was configured incorrectly.
  :jira:`3004`

- Removed some memory leaks from version 2 of the Python bindings.
  :jira:`2981`

- Fixed a bug introduced in HTCondor 24.7.0 which would lead to the directory
  ``0`` (and subdirectories) spuriously being created in the :macro:`SPOOL`
  directory.
  :jira:`3026`

*** 24.0.8 bugs

- Fixed a bug introduced in 24.0.7 and 24.7.3 when running on Linux cgroup v1
  systems, jobs that were killed by the out-of-memory killer were
  considered completed instead of being put on hold.
  :jira:`3094`

- The :class:`htcondor2.Credd` initializer now properly raised
  a :class:`TypeError` if the location argument isn't
  a :class:`classad2.ClassAd`, rather than failing to raise a ``TypError``.
  :jira:`2993`

*** 24.7.3 features

- Improved the ability of :tool:`condor_who` to query *condor_startd* processes when
  *condor_who* is running as root or as the same user as the Startd, and added
  formatting options for use when the *condor_startd* is running as a job on
  another batch system.
  :jira:`2927`

- :tool:`htcondor credential add oauth2` can now be used to store tokens that can
  be used by jobs via :subcom:`use_oauth_services`. The user is responsible for
  updating tokens that can expire.
  :jira:`2803`

- Added :ad-attr:`OSHomeDir` to starter's copy of the job ad.
  :jira:`2972`

- Add :macro:`SYSTEM_MAX_RELEASES` which implements an upper bound on the number
  of times any job can be released by a user or periodic expression.
  :jira:`2926`

- Added the ability for an EP administrator to disable access to the network
  by a job, by setting :macro:`NO_JOB_NETWORKING` to true.
  :jira:`2967`

- Added the ability for a docker universe job to fetch an authenticated
  image from the docker repository.
  :jira:`2870`

- Improved :tool:`condor_watch_q` to display information about the number of
  jobs actively transferring input or output files.
  :jira:`2958`

- The default value for :macro:`DISABLE_SWAP_FOR_JOB` has been changed to
  ``True``.  This provides a more predictable and uniform user experience
  for jobs running on different EPs.
  :jira:`2960`

- Add :tool:`htcondor annex login` verb, which opens a shared SSH connection to
  the named HPC system.  If you've recently created or added an annex at a
  particular system, it will re-use that cached connection; otherwise, you'll have
  to login again, but that connection will then be re-usable by other
  :tool:`htcondor annex` commands.
  :jira:`2809`

- Updated :tool:`htcondor annex` to work with Expanse's new requirements for its
  ``gpu`` and ``gpu-shared`` queues.
  :jira:`2634`

- Enhanced :tool:`htcondor job status` to also show the time to transfer the
  job input sandbox.
  :jira:`2959`

- Jobs that use :subcom:`concurrency_limits` can now re-use claims in
  the schedd.
  :jira:`2937`

- Added :subcom:`shell` for Linux systems.
  :jira:`2918`

- :macro:`START_VANILLA_UNIVERSE` expressions may now refer to attributes
  in the schedd add using the prefix ``SCHEDD``.
  :jira:`2919`

- Hold messages generated by failure to transfer output now include how many
  files failed to transfer.
  :jira:`2903`

- Added ``-transfer-history`` flag to :tool:`condor_history` to query historical
  Input, Output, and Checkpoint transfer ClassAds stored in the :macro:`JOB_EPOCH_HISTORY`
  files.
  :jira:`2878`

- Improved the parsing and handling of syntax errors in the
  :subcom:`transfer_output_remaps` submit command.
  :jira:`2920`

- DAGMan :dag-cmd:`SERVICE` nodes will no longer be removed automatically when
  a DAG contains a :dag-cmd:`FINAL` node and :tool:`condor_rm` is used on the
  DAGMan scheduler job.
  :jira:`2938`

- The list of files generated by the :subcom:`manifest` submit command now
  recursively includes subdirectories.
  :jira:`2903`

- Added new option ``-extract`` to :tool:`condor_history` to copy historical
  ClassAd entries that match a provided constraint to a specified file.
  :jira:`2923`

- EPs using disk enforcement via LVM and :macro:`LVM_HIDE_MOUNT` = ``True``
  will now advertise :ad-attr:`HasVM` = ``False`` due to VM universe jobs being
  incompatible with mount namespaces.
  :jira:`2945`

- Added support for running Docker universe on ARM hosts
  :jira:`2906`

- The **CLAIMTOBE** authentication protocol now fully qualified user names
  with the system's ``$(UID_DOMAIN)``.  To revert to the former semantics,
  set :macro:`SEC_CLAIMTOBE_INCLUDE_DOMAIN` to false.
  :jira:`2915`

*** 24.7.3 bugs

- Fixed a bug in the local issuer credential monitor that prevented the issuance of tokens
  using the WLCG profile.
  :jira:`2954`

- Fixed bug where DAGMan would output an error message containing garbage
  when dumping failed node information to the debug log.
  :jira:`2899`

- Fixed a bug where EP's using :macro:`STARTD_ENFORCE_DISK_LIMITS` would mark a
  slot as ``broken`` when the *condor_starter* fails to remove the ephemeral logical
  volume but the *condor_startd* successfully removes the LV.
  :jira:`2953`

- Fixed a bug in the Vault credential monitor that kept credentials from being fetched
  if VAULT_CREDMON_PROVIDER_NAMES was unset. Introduced in HTCondor 24.3.0.
  :jira:`2912`

- Fixed a bug in the local issuer credential monitor that kept credentials from being
  issued if :macro:`LOCAL_CREDMON_TOKEN_VERSION` (or named variant) was not set.
  :jira:`2965`

*** 24.0.7 features

- The *condor_startd* now distributes the :ad-attr:`LoadAvg` assigned to a partitionable slot
  to the idle resources of the partitionable slot, and then to the dynamic slots.
  Machines that have only a single partitionable slot will now have the same behavior under
  a :macro:`use policy:DESKTOP` as they did in version 23.10.18 and 24.0.1.
  :jira:`2901`

*** 24.0.7 bugs

- When using delegated cgroup v2, HTCondor no longer reports that that main job (often a pilot)
  has an out of memory condition when only the sub-job has hit an oom.
  :jira:`2944`

*** 24.5.2 bugs

- The default value for :macro:`STARTD_LEFTOVER_PROCS_BREAK_SLOTS` has been
  changed to 'False'. When 'True', the EP was erroneously marking slots as
  broken.
  :jira:`2946`

*** 24.5.1 features

- The *condor_starter* now advertise :ad-attr:`StdoutMtime` and :ad-attr:`StderrMtime`
  which represent the most recent modification time, in seconds since the epoch
  of a job which uses file transfer.
  :jira:`2837`

- The *condor_startd*, when running on a machine with Nvidia gpus, now advertises
  Nvidia driver version.
  :jira:`2856`

- Increased the default width of :tool:`condor_qusers` output when redirected to a
  file or piped to another command to prevent truncation.
  :jira:`2861`

- The *condor_startd* will now never lose track and leak logical volumes that were failed
  to be cleaned up when using :macro:`STARTD_ENFORCE_DISK_LIMITS`. The *condor_startd*
  will now periodically retry removal of logical volumes with an exponential back off.
  :jira:`2852`

- The *condor_startd* will now keep dynamic slots that have a :ad-attr:`SlotBrokenReason`
  attribute in ``Unclaimed`` state rather than deleting them when they change state
  to ``Unclaimed``.  A new configuration variable :macro:`CONTINUE_TO_ADVERTISE_BROKEN_DYNAMIC_SLOTS`
  controls this behavior.  It defaults to ``true`` but can be set to ``false`` to preserve
  the old behavior.  This change also adds a new attribute :ad-attr:`BrokenContextAds`
  to the daemon ad of the *condor_startd*. This attribute has a ClassAd for each broken resource
  in the startd.  *condor_status* has been enhanced to use this new attribute to display
  more information about the context of broken resources when both ``-startd`` and ``-broken``
  arguments are used.
  :jira:`2844`

- The *condor_startd* will now permanently reduce the total slot resources advertised by
  a partitionable slot when a dynamic slot is deleted while it is marked as broken. The
  amount of reduction will be advertised in new attributes such as ad-attr:`BrokenSlotCpus`
  so that the original size of the slot can be computed.
  :jira:`2865`

- Daemons will now more quickly discover with a non-responsive
  *condor_collector* has recovered and resume advertising to it.
  :jira:`2605`

- Jobs can now request user credentials generated by any combination of the
  OAuth2, Local Issuer, and Vault credential monitors on the AP.
  Remote submitters can request these credentials without having any of the
  CREDMON-related parameters in their configuration files.
  :jira:`2851`

*** 24.5.1 bugs

- Fixed a bug where the *condor_gridmanager* would write to log file
  `GridmanagerLog.root` after a reconfiguration.
  :jira:`2846`

- ``htcondor annex shutdown`` now works again.
  :jira:`2808`

- Fixed a bug where the job state table DAGMan prints to its debug file could
  contain a negative number for the count of failed jobs.
  :jira:`2872`

*** 24.0.5 bugs

*** 24.4.0 features

- Improved validation and cleanup of :macro:`EXECUTE` directories.
  The :macro:`EXECUTE` directory must now be owned by the *condor* user
  when the daemons are started as root. The condor_startd will
  not attempt to clean an invalid :macro:`EXECUTE` directory nor will
  it alter the file permissions of an :macro:`EXECUTE` directory.
  :jira:`2789`

- For **batch** grid :subcom:`universe` jobs, the PATH environment variable values
  from the job ad and the worker node environment are now combined.
  Previously, only the PATH value from the job ad was used.
  The old behavior can be restored by setting ``blah_merge_paths=no`` in
  the ``blah.config`` file.
  :jira:`2793`

- Many small improvements to :tool:`condor_q` ``-analyze`` and ``-better-analyze``
  for pools that use partitionable slots.  As a part of this, the *condor_schedd*
  was changed to provide match information for the auto-cluster of
  the job being analyzed, which :tool:`condor_q` will report if it is available.
  :jira:`2720`

- The *condor_startd* now advertises a new attribute,
  :ad-attr:`SingularityUserNamespaces` which is ``true`` when Apptainer
  or Singularity work and are using Linux user namespaces, and ``false``
  when it is using setuid mode.
  :jira:`2818`

- The *condor_startd* daemon ad now contains attributes showing the average and total
  bytes transferred to and from jobs during its lifetime.
  :jira:`2721`

- The *condor_credd* daemon no longer listens on port ``9620`` by default,
  but rather uses the *condor_shared_port* daemon.
  :jira:`2763`

- DAGMan will now periodically print a table regarding states of
  job placed to the Access Point to the debug log (``*.dagman.out``).
  The rate at which this table in printed is dictated by
  :macro:`DAGMAN_PRINT_JOB_TABLE_INTERVAL`
  :jira:`2794`

*** 24.4.0 bugs

- Changed the numeric output of :tool:`htcondor job status` so that the rounding
  to megabytes, gigabytes, etc. matches the binary definitions the rest
  of the tools use.
  :jira:`2788`

*** 24.0.4 features

- For **arc** grid :subcom:`universe` jobs, the new submit command
  :subcom:`arc_data_staging` can be used to supply additional elements
  to the DataStaging block of the ARC ADL that HTCondor constructs.
  :jira:`2774`

*** 24.0.4 bugs

- Fixed a bug in the negotiator that caused it to crash when matching
  offline ads.
  :jira:`2819`

- Fixed a memory leak in the schedd that could be caused by ``SCHEDD_CRON``
  scripts that generate standard error output.
  :jira:`2817`

- Fixed a bug that cause the *condor_schedd* to crash with a segmentation
  fault if a :tool:`condor_off` ``-fast`` command was run while a schedd cron
  script was running.
  :jira:`2815`
