*** 25.8.2 features

- HTCondor now correctly detects and advertises the NVIDIA driver version
  for NVIDIA GPUs which are configured to run in "MIG" mode.
  :jira:`3626`

- Singularity or Apptainer jobs now also pass ``--rocm`` to GPU jobs
  to support AMD jobs, unless :macro:`SINGULARITY_ADD_ROCM_FLAG` is
  set to false.
  :jira:`3537`

- Added a new ``-transfer`` option to :tool:`condor_rm`.  When used to remove
  a running job, the job will transfer any output files it has produced
  back to the submit machine before exiting.
  :jira:`3519`

- :tool:`condor_q` now supports a ``-aaf`` option
  that works like ``-autoformat`` (``-af``) but appends the specified
  attributes as additional columns after the standard output columns,
  rather than replacing them.
  :jira:`3571`

- :tool:`condor_top` now supports a ``-history`` option to read the most
  recently recorded on disk performance statistics.  Runs much faster,
  and works even when the *condor_schedd* is slow or not running.
  :jira:`3582`

- :tool:`htcondor dag status` now also shows information about jobs
  transferring sandboxes and in the transfer queue.
  :jira:`3517`

- :tool:`htcondor job status` and :tool:`condor_q` now indicate when a job is
  in the cooldown state.
  :jira:`3525`

- :tool:`condor_run` no longer passes :jdl:`getenv = true` to the
  generated submit file.  There is a new option ``-getenv`` to 
  allow this.
  :jira:`3549`

- :tool:`condor_ssh_to_job` now work with container universe jobs where
  the Singularity or Apptainer runtime uses the setuid flow.
  :jira:`3308`

- The negotiator now has access to the :ad-attr:`RemoteUserFloor` to avoid
  going beneath a user's floor number of resources while preempting.
  :jira:`3226`

- The *condor_shadow* can now log a CSV record to the :macro:`SHADOW_RECONNECT_LOG`
  file each time it successfully reconnects to a disconnected starter or
  gives up trying.  Each record contains timing information about when the
  job was activated, the last known contact time, whether the reconnect
  succeeded, the configured lease duration, and whether the starter is
  known to be dead.  This is controlled by :macro:`SHADOW_LOG_RECONNECT`
  which defaults to ``True``.
  :jira:`3593`

- :tool:`condor_config_val` now has a -default option to shows the default
  value of a knob.
  :jira:`2683`

- Jobs that were submitted with spooled data files and have completed will
  now leave the queue after a successful retrieval of their output files
  (e.g. with :tool:`condor_transfer_data`).
  :jira:`3508`

- Failed URL transfer where the URL was longer than about 100 bytes would
  never have the URL scheme logged in the hold message or user log.
  :jira:`3412`

- All of the :tool:`htcondor` sub-tools (which are written in Python)
  support python 3.12.
  :jira:`2213`

- In the python bindings, functions arguments that used to have a default
  of the :obj:`None` type now take a python Optional.  This is more compliant
  with PEP 484, and is more in line with what python linting tools
  expect.  There should be no changes needed for clients of the python
  bindings.
  :jira:`3553`

- When cgroups are active, *condor_startds* now advertise
  :ad-attr:`CgroupEnforced` to be true.
  :jira:`3555`

- Enabling/disabling AP users now requires ADMINISTRATOR authorization.
  Previously, WRITE with queue superuser authorization was required.
  :jira:`3550`

- HTCondor now more quickly detects when the network connection between
  two daemons is broken or disconnected.
  :jira:`3547`

- Updated list of default environment variables that *condor_dagman* always
  grabs from the users environment to **not** include ``XDG_RUNTIME_DIR``.
  :jira:`1913`

- :tool:`condor_status` ``-server`` now correctly labels the count of discovered Slots
  in the summary table.
  :jira:`3535`

- When placing a job with credentials, the client will now find the
  address of the *condor_credd* via the *condor_schedd* when possible.
  The client already did this for Kerberos credentials; it now does so
  for all job credential types.
  :jira:`3536`

- Added new ``histogram`` option to :tool:`htcondor eventlog` and :tool:`htcondor dag`.
  :jira:`3530`

- The **copy_to_spool** command in :tool:`condor_submit` is no longer
  supported. This does not affect the ability to spool a job's full input sandbox.
  :jira:`3545`

- No longer warn about configuration knobs with dots in the names if the knob
  is group quota related, and begins with GROUP.
  :jira:`3544`

- The old ClassAd syntax is no longer supported in job transforms.
  :jira:`3538`

- Improved DAGMan debugging when to print warnings produced by *condor_schedd*\'s
  :ref:`submit-requirements`.
  :jira:`3573`

- Removed deprecated full rescue file for DAGMan. With this comes the removal of
  the configuration option ``DAGMAN_WRITE_PARTIAL_RESCUE`` and the command line
  option ``-DumpRescue`` for :tool:`condor_submit_dag`.  This has been deprecated since
  2011, so there should be no problems with compatibility.
  :jira:`3565`

- Improved DAGMan's debugging error when the provisioner node exits prior to notifying
  DAGMan of successful resource provisioning.
  :jira:`3542`

- Added new configuration parameter :macro:`SEC_USE_LOW_DATA_MODE`.
  When set to ``True``, daemons will reject large data messages from
  unauthenticated clients.
  The default is ``False``.
  :jira:`3393`

- The :macro:`VM_NETWORKING_TYPE` knob is now case insensitive,
  matching the documentation.
  :jira:`3557`

- Bootstrap CSS framework used in OAuth Credmon login page updated
  to version 5.3.8.
  :jira:`3576`

*** 25.8.2 bugs

- Fixed bugs where a job would be placed on hold if there was a problem transferring
  files via Pelican (OSDF) or HTTP.  Now in several cases where the problem was the
  fault of HTCSS and not the fault of the job, the job will go back to idle and attempt
  to be rematched instead of going on hold.  Several additional VacateReasonSubCodes where
  added in support of this; see section :ref:`Vacate Reason Codes`.
  :jira:`2952`

- Fixed a bug that caused the *condor_starter* to log confusing errors and not exit
  promptly if it failed to launch a Singularity job.
  :jira:`3531`

- Fixed a bug in container universe with :subcom:`container_target_dir` when file transfer
  was off, that caused jobs to start in the users home directory.
  :jira:`3575`

- Fixed a minor bug whereby the Access Point is unaware that the Execution Point relinquishes a
  match when the EP's :macro:`WANT_HOLD` is triggered.  This can result in the AP failing to
  activate the next job in a job list due to VacateReasonCode 1033 ``ActivationRefusedUnclaimed``.
  :jira:`3609`

*** 25.0.9 bugs

*** 25.7.2 features

- Jobs whose ClassAds are larger than 64 kilobytes long caused excessive load on the
  *condor_schedd*. The most common case of such a very large ClassAd is the use
  of :subcom:`getenv`.  This could cause the :ad-attr:`RecentDaemonCoreDutyCycle` to
  spike to near 100% for hours, and cause :tool:`condor_submit` to timeout and
  fail. Pipe buffers in the *condor_schedd* are now appropriately sized to fix
  this problem.
  :jira:`3507`

- Improved :tool:`condor_watch_q` to sort rows of data display rows with jobs still
  active in the queue before rows of jobs that have all exited the AP job queue.
  :jira:`3516`

- Added new command line options to :tool:`condor_watch_q` to control
  displaying rows of jobs that have all exited the AP job queue.
  :jira:`3516`

- Added ``-interval`` command line option to :tool:`condor_watch_q` to control the
  refresh rate of the tool.
  :jira:`3528`

- When a job exceeds its memory request, instead of reporting the memory 
  measured in the last polling period, the error message reports the 
  size of the provisioned memory to more accurately report what happened.
  :jira:`3514`

- All file transfer records written to the :macro:`JOB_EPOCH_HISTORY` archive
  will contain :ad-attr:`Owner` and :ad-attr:`RemoteHost` copied from the associated
  job record.
  :jira:`3497`

- Added new :meth:`htcondor2.Startd.vacateSlot` and :meth:`htcondor2.Startd.vacate`
  methods to the python API to do equivalent actions as :tool:`condor_vacate`.
  :jira:`3449`

- Added support for submitting jobs to the Flux job management system via the
  grid universe **batch** type.
  :jira:`3500`

*** 25.7.2 bugs

*** 25.0.8 bugs

*** 25.6.1 features

- DAGMan now uses the new DAG file parser introduced for :tool:`condor_dag_checker`. The old
  parser will be deprecated in the future but can be used by setting :macro:`DAGMAN_USE_OLD_FILE_PARSER`
  to ``True``.
  :jira:`3304`

- Augment the *condor_kbdd* to use the systemd sd-login library to find user
  sessions, since /usr/run/utmp is no longer available on newer platforms.
  Also, add idle check support for Mutter(Gnome), which works on both
  Wayland and X11.
  :jira:`3452`

- Added new ``-RescueFile`` option to :tool:`condor_submit_dag` to specify a
  DAGMan rescue file by name for state restoration.
  :jira:`3425`

- The submit knob :macro:`SUBMIT_REQUEST_MISSING_UNITS` now also applies to
  :subcom:`retry_request_memory` as well.
  :jira:`3419`

- Added :meth:`~classad2.ClassAd.formatJson` and :meth:`~classad2.ClassAd.formatOld` as
  aliases for :meth:`~classad2.ClassAd.printJson` and :meth:`~classad2.ClassAd.printOld`,
  respectively.
  :jira:`3341`

- If a docker universe jobs fails on an execution point because the image
  was built for a machine architecture that does not match the execution
  point, the hold message now more clearly indicates why.
  :jira:`3447`

- When the attempt to claim a slot to run a job is refused, the reason for the
  refusal is now reported back to the *condor_shadow* which logs the reason and
  adds the reason code to the :ad-attr:`NumVacatesByReason` attribute of the job.
  :jira:`3388`

- The option for the *condor_schedd* to keep job event log files open for
  a period of time has been removed.
  This was controlled by the now-removed configuration parameters
  USERLOG_FILE_CACHE_MAX and USERLOG_FILE_CACHE_CLEAR_INTERVAL.
  It was disabled by default.
  :jira:`3367`

- By default, HTCondor will now send extra information to "multi-file" file
  transfer plug-ins if such is specified in the job.  All such plug-ins
  should safely ignore this extra information.  To ease the transition if a
  plug-in does not, the extra information may be disabled by unsetting
  :macro:`ASSUME_COMPATIBLE_MULTIFILE_PLUGINS` on the relevant EPs.  (We
  have deliberately not documentation the format of the extra information
  because this feature is under active development.)

  Plug-in authors may find the
  :doc:`/apis/file-transfer-plugins/index` documentation helpful.
  :jira:`3357`

*** 25.6.1 bugs

- Fixed a bug when STARTER_NESTED_SCRATCH was true that broke the
  :subcom:`starter_log` submit command.
  :jira:`3408`

- The EPs produced by :tool:`htcondor annex` now include the additional
  configuration distributed with the HTCondor tarball, which should make
  them less sensitive to version changes.  Additionally, the default
  configuration for HTCondor now enables the Pelican (OSDF) file transfer
  plug-in.
  :jira:`3410`

- Fixed a bug where :tool:`condor_ssh_to_job` when running into a
  Singularity or Apptainer enabled job with :macro:`SINGULARITY_TARGET_DIR`
  set to true would start out with the current directory one higher than
  the job.
  :jira:`3467`

- Fixed a bug in the handling of :subcom:`retry_request_memory` where a
  list of values using a scale such as ``GB`` would generate an error
  or fail quietly if the ``B`` was not present.
  :jira:`3458`

- Fixed an intermittent bug where the *condor_shadow* would occasionally
  get stuck for 20 minutes trying to reconnect to a non-existent *condor_starter*
  when file transfer file fails for any reason.
  :jira:`3429`

*** 25.0.6 features

- Initial support for openSUSE Leap 16.
  :jira:`3233`

- Strip out VOMS and EC2 GAHP support in the HTCondor Python wheels, so that
  the HTCondor Python wheel is usable with the python-slim Docker image.
  :jira:`3469`

*** 25.0.6 bugs

*** 25.5.1 features

- A new configuration variable :macro:`FORCE_NEGOTIATOR_SLOT_WEIGHT` allows
  an administrator to configure the *condor_negotiator* so it uses its
  own expression of :macro:`SLOT_WEIGHT` rather than allowing the execution
  point to declare the weight of a slot.
  :jira:`3346`

- :macro:`LVM_CLEANUP_FAILURE_MAKES_BROKEN_SLOT` now defaults to ``True``\,
  causing failure to cleanup ephemeral Logical Volumes to break slot resources.
  If the logical volume is later successfully cleaned up then the associated
  broken resources will be restored.
  :jira:`3352`

- Added :ad-attr:`CleanupCategoryCounts` to record the number of cleanup
  reminders a *condor_startd* is tracking.
  :jira:`3352`

- The *condor_negotiator* will now apply the :macro:`REMOVE_SIGNIFICANT_ATTRIBUTES`
  list to the attributes it will be sending to the *condor_schedd* for resource requests.
  The *condor_negotiator* now also has a more complete set of known slot attributes which it
  will remove as well.  These changes should help make the *condor_schedd* more efficient when building
  resource requests for matchmaking.
  :jira:`3358`

- EP administrators may now alter the semantics of self-checkpointing jobs.
  After a job self-checkpoints and the checkpoint is uploaded, if
  :macro:`CHECK_REACTIVATE_AFTER_CHECKPOINT` is set, the EP will check if the slot could
  be reactivated then the job is restarted: otherwise the job is evicted with a
  :doc:`vacate reason <../codes-other-values/vacate-reason-codes>`.
  :jira:`3222`

- Fix up the *condor_credmon_oauth* to support Python 3.14.  With this, we believe that
  all of HTCondor will work with Python 3.14.
  :jira:`3365`

- Added configuration variables :macro:`JOB_ROUTER_SCHEDD1_ADDRESS_FILE` and
  :macro:`JOB_ROUTER_SCHEDD2_ADDRESS_FILE`. When configured, the *condor_job_router*
  will first use the address file to locate the source or destination *condor_schedd* and
  only query the *condor_collector* if the file does not contain an address. Use of these
  configuration variables improves the efficiency of the job router and eliminates
  one source of failure when routing.
  :jira:`1736`

- Removed the ``NAMED_CHROOT`` configuration parameter for the
  *condor_starter*. It was an inferior form of containerization that was
  never fully supported.
  :jira:`3380`

- Improved DAGMan's debugging for executed scripts.
  :jira:`3381`

*** 25.5.1 bugs

*** 25.0.5 features

- Initial support for Ubuntu 24.04 (noble numbat) on the arm64 platform.
  :jira:`3414`

*** 25.0.5 bugs

*** 25.4.0 features

- Changed the default value of :macro:`STARTER_NESTED_SCRATCH` to true.  When false,
  the job scratch directory (named ``dir_XXX``) is the immediate child of the :macro:`EXECUTE`
  directory.  When true, the job's scratch directory is a sub-directory thereof, named
  ``scratch``, with the intention that all condor metadata files will be moved to peer
  directories, and not pollute the job's scratch directory with condor control files.
  :jira:`3080`

- The *condor_starter* can now refresh job credentials and respond to
  commands while doing output file transfers.
  :jira:`3259`

- When a job has a :subcom:`transfer_output_remaps` that maps to an
  absolute path, HTCondor will now create any missing intermediate
  directories in that path, if allowed by the user's permissions.
  :jira:`3318`

- :tool:`condor_watch_q` will now display for total number of jobs for
  clusters using late materialization.
  :jira:`3164`

- The *condor_schedd* can now limit the number of running jobs on a per-user basis.
  The *condor_schedd* will not allow more jobs than the :ad-attr:`MaxJobsRunning` in
  the user record to begin running.
  :jira:`3325`

- Added new job ClassAd attributes :ad-attr:`NumVacatesPreExecution`
  and :ad-attr:`NumVacatesByReasonPreExecution`. These attributes provide
  counts about why a job was kicked off of an Execution Point before the
  job was actually spawned (i.e. due to failure to transfer input files,
  failure to setup the job environment, etc).
  :jira:`3339`

- The *condor_startd* will now provision :ad-attr:`Disk` to slots as a specific quantity in KiB
  rather than as a percentage of free disk space.  :ad-attr:`Disk` provisioned to dynamic slots
  will now be much closer in size to the request. Provisioning statements in the
  configuration files are no longer required to be expressed as a percentage, but a percentage
  will still work as before.  The *condor_startd* now reports both :ad-attr:`TotalDisk`
  and :ad-attr:`DetectedDisk` in each slot and in the daemon ad.
  :jira:`3302`

- Added new Python binding functions to reload the ClassAd usermaps and to load
  a specific usermap for use with the ``userMap`` ClassAd function.
  :jira:`3274`

- Docker universe jobs now correctly report CPU and memory usage when the
  Docker server is installed in a non-standard place, pointed at with
  the ``DOCKER_HOST`` environment variable, such as with a docker-in-docker setup.
  :jira:`3280`

- More accurately report the reason why a job's execution was interrupted
  (via the evict log event and :ad-attr:`VacateReason` job attribute).
  :jira:`3295`

- Removed option to have the *condor_schedd* send claim lease keep-alives.
  Now, the *condor_startd* always sends the keep-alives, which was the
  default.
  :jira:`3312`

- For **batch** grid :subcom:`universe` jobs, the ``PATH`` environment variable
  is now used by default to locate the tools of the destination batch scheduler.
  You can specify a specific directory where the tools are located by setting
  the appropriate binpath variable in the ``blah.config`` file (e.g. ``slurm_binpath=/opt/slurm/bin``).
  :jira:`3326`

*** 25.4.0 bugs

*** 25.3.1 bugs

*** 25.0.3 bugs

*** 25.2.1 features

- A job can now specify both an initial value for :subcom:`request_memory`
  and a retry with a larger memory request if the job uses more memory than the initial request.
  The default value for :subcom:`request_memory` and for :subcom:`request_disk`
  have been changed so that jobs will no longer default to trying to grow the request
  based on previous usage.
  :jira:`3227`

- Improved DAGMan to allow :dag-cmd:`SUBMIT-DESCRIPTION`\'s to be declared
  after nodes that reference the named inline submit description.
  :jira:`3230`

- DAGMan will now output a structured message for job submission failures
  to the debug log file. This message is formatted similarly to the read
  userlog job event message.
  :jira:`3257`

- Added new DAGMan Post :dag-cmd:`SCRIPT` macro to provide exit code for each
  job associated with a node. This is in the form of a comma separated list.
  :jira:`3245`

- The *condor_credmon_local* can now use RSA-based (RS256) keys to generate
  tokens by setting :macro:`LOCAL_CREDMON_PRIVATE_KEY_ALGORITHM` to ``RS256``.
  :jira:`3266`

- Added the ability to set a ClassAd expression in
  :macro:`LOCAL_CREDMON_AUTHZ_TEMPLATE_EXPR` to define scopes for tokens
  generated by the *condor_credmon_local*.
  :jira:`3261`

- Added new fine grained attributes :ad-attr:`NumPendingClaimsInQueue` and
  :ad-attr:`NumPendingClaimsInContact` to the *condor_schedd* record.
  :jira:`3125`

*** 25.2.1 bugs

- Fixed a problem where setting :macro:`MAX_FILE_TRANSFER_PLUGIN_LIFETIME`
  to longer than :macro:`STARTER_FILE_XFER_STALL_TIMEOUT` would prevent the
  logging of output for hung file-transfer plug-ins.
  :jira:`3202`

*** 25.0.2 bugs

- Fixed a bug causing some file-transfer plug-ins (box, gdrive, and onedrive)
  to fail immediately on start-up.
  :jira:`3317`

- Fixed bug where when :macro:`SINGULARITY_TARGET_DIR` was set and
  :macro:`STARTER_NESTED_SCRATCH` was also set, then some environment
  variables that pointed into the job's execute directory were not correct.
  :jira:`3265`

*** 25.1.0 features

*** 25.1.0 bugs

*** 25.0.1 features

- New implementations of the Python bindings, ``htcondor2`` and ``classad2``. This
  re-implementation of the Python bindings fixes short-comings in the original
  implementation and eases future maintenance.
  :jira:`2153`

- The original implementation of the Python bindings has been removed.
  To use the new Python bindings, please read the
  :doc:`migration guide <../apis/python-bindings/api/version2/migration-guide>`.
  :jira:`3240`

*** 25.0.1 bugs

- Fixed a bug where *condor_qusers* would treat the ``add`` option as ``enable``
  when the leading dash was omitted.
  :jira:`3284`

