Skip to content
This repository was archived by the owner on Apr 23, 2020. It is now read-only.

Commit 8f85685

Browse files
committed
Enhance synchscope representation
OpenCL 2.0 introduces the notion of memory scopes in atomic operations to global and local memory. These scopes restrict how synchronization is achieved, which can result in improved performance. This change extends existing notion of synchronization scopes in LLVM to support arbitrary scopes expressed as target-specific strings, in addition to the already defined scopes (single thread, system). The LLVM IR and MIR syntax for expressing synchronization scopes has changed to use *syncscope("<scope>")*, where <scope> can be "singlethread" (this replaces *singlethread* keyword), or a target-specific name. As before, if the scope is not specified, it defaults to CrossThread/System scope. Implementation details: - Mapping from synchronization scope name/string to synchronization scope id is stored in LLVM context; - CrossThread/System and SingleThread scopes are pre-defined to efficiently check for known scopes without comparing strings; - Synchronization scope names are stored in SYNC_SCOPE_NAMES_BLOCK in the bitcode. Differential Revision: https://reviews.llvm.org/D21723 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@307722 91177308-0d34-0410-b5e6-96231b3b80d8
1 parent fdda7ea commit 8f85685

File tree

69 files changed

+1227
-791
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

69 files changed

+1227
-791
lines changed

docs/LangRef.rst

+45-35
Original file line numberDiff line numberDiff line change
@@ -2209,12 +2209,21 @@ For a simpler introduction to the ordering constraints, see the
22092209
same address in this global order. This corresponds to the C++0x/C1x
22102210
``memory_order_seq_cst`` and Java volatile.
22112211

2212-
.. _singlethread:
2212+
.. _syncscope:
22132213

2214-
If an atomic operation is marked ``singlethread``, it only *synchronizes
2215-
with* or participates in modification and seq\_cst total orderings with
2216-
other operations running in the same thread (for example, in signal
2217-
handlers).
2214+
If an atomic operation is marked ``syncscope("singlethread")``, it only
2215+
*synchronizes with* and only participates in the seq\_cst total orderings of
2216+
other operations running in the same thread (for example, in signal handlers).
2217+
2218+
If an atomic operation is marked ``syncscope("<target-scope>")``, where
2219+
``<target-scope>`` is a target specific synchronization scope, then it is target
2220+
dependent if it *synchronizes with* and participates in the seq\_cst total
2221+
orderings of other operations.
2222+
2223+
Otherwise, an atomic operation that is not marked ``syncscope("singlethread")``
2224+
or ``syncscope("<target-scope>")`` *synchronizes with* and participates in the
2225+
seq\_cst total orderings of other operations that are not marked
2226+
``syncscope("singlethread")`` or ``syncscope("<target-scope>")``.
22182227

22192228
.. _fastmath:
22202229

@@ -7380,7 +7389,7 @@ Syntax:
73807389
::
73817390

73827391
<result> = load [volatile] <ty>, <ty>* <pointer>[, align <alignment>][, !nontemporal !<index>][, !invariant.load !<index>][, !invariant.group !<index>][, !nonnull !<index>][, !dereferenceable !<deref_bytes_node>][, !dereferenceable_or_null !<deref_bytes_node>][, !align !<align_node>]
7383-
<result> = load atomic [volatile] <ty>, <ty>* <pointer> [singlethread] <ordering>, align <alignment> [, !invariant.group !<index>]
7392+
<result> = load atomic [volatile] <ty>, <ty>* <pointer> [syncscope("<target-scope>")] <ordering>, align <alignment> [, !invariant.group !<index>]
73847393
!<index> = !{ i32 1 }
73857394
!<deref_bytes_node> = !{i64 <dereferenceable_bytes>}
73867395
!<align_node> = !{ i64 <value_alignment> }
@@ -7401,14 +7410,14 @@ modify the number or order of execution of this ``load`` with other
74017410
:ref:`volatile operations <volatile>`.
74027411

74037412
If the ``load`` is marked as ``atomic``, it takes an extra :ref:`ordering
7404-
<ordering>` and optional ``singlethread`` argument. The ``release`` and
7405-
``acq_rel`` orderings are not valid on ``load`` instructions. Atomic loads
7406-
produce :ref:`defined <memmodel>` results when they may see multiple atomic
7407-
stores. The type of the pointee must be an integer, pointer, or floating-point
7408-
type whose bit width is a power of two greater than or equal to eight and less
7409-
than or equal to a target-specific size limit. ``align`` must be explicitly
7410-
specified on atomic loads, and the load has undefined behavior if the alignment
7411-
is not set to a value which is at least the size in bytes of the
7413+
<ordering>` and optional ``syncscope("<target-scope>")`` argument. The
7414+
``release`` and ``acq_rel`` orderings are not valid on ``load`` instructions.
7415+
Atomic loads produce :ref:`defined <memmodel>` results when they may see
7416+
multiple atomic stores. The type of the pointee must be an integer, pointer, or
7417+
floating-point type whose bit width is a power of two greater than or equal to
7418+
eight and less than or equal to a target-specific size limit. ``align`` must be
7419+
explicitly specified on atomic loads, and the load has undefined behavior if the
7420+
alignment is not set to a value which is at least the size in bytes of the
74127421
pointee. ``!nontemporal`` does not have any defined semantics for atomic loads.
74137422

74147423
The optional constant ``align`` argument specifies the alignment of the
@@ -7509,7 +7518,7 @@ Syntax:
75097518
::
75107519

75117520
store [volatile] <ty> <value>, <ty>* <pointer>[, align <alignment>][, !nontemporal !<index>][, !invariant.group !<index>] ; yields void
7512-
store atomic [volatile] <ty> <value>, <ty>* <pointer> [singlethread] <ordering>, align <alignment> [, !invariant.group !<index>] ; yields void
7521+
store atomic [volatile] <ty> <value>, <ty>* <pointer> [syncscope("<target-scope>")] <ordering>, align <alignment> [, !invariant.group !<index>] ; yields void
75137522

75147523
Overview:
75157524
"""""""""
@@ -7529,14 +7538,14 @@ allowed to modify the number or order of execution of this ``store`` with other
75297538
structural type <t_opaque>`) can be stored.
75307539

75317540
If the ``store`` is marked as ``atomic``, it takes an extra :ref:`ordering
7532-
<ordering>` and optional ``singlethread`` argument. The ``acquire`` and
7533-
``acq_rel`` orderings aren't valid on ``store`` instructions. Atomic loads
7534-
produce :ref:`defined <memmodel>` results when they may see multiple atomic
7535-
stores. The type of the pointee must be an integer, pointer, or floating-point
7536-
type whose bit width is a power of two greater than or equal to eight and less
7537-
than or equal to a target-specific size limit. ``align`` must be explicitly
7538-
specified on atomic stores, and the store has undefined behavior if the
7539-
alignment is not set to a value which is at least the size in bytes of the
7541+
<ordering>` and optional ``syncscope("<target-scope>")`` argument. The
7542+
``acquire`` and ``acq_rel`` orderings aren't valid on ``store`` instructions.
7543+
Atomic loads produce :ref:`defined <memmodel>` results when they may see
7544+
multiple atomic stores. The type of the pointee must be an integer, pointer, or
7545+
floating-point type whose bit width is a power of two greater than or equal to
7546+
eight and less than or equal to a target-specific size limit. ``align`` must be
7547+
explicitly specified on atomic stores, and the store has undefined behavior if
7548+
the alignment is not set to a value which is at least the size in bytes of the
75407549
pointee. ``!nontemporal`` does not have any defined semantics for atomic stores.
75417550

75427551
The optional constant ``align`` argument specifies the alignment of the
@@ -7597,7 +7606,7 @@ Syntax:
75977606

75987607
::
75997608

7600-
fence [singlethread] <ordering> ; yields void
7609+
fence [syncscope("<target-scope>")] <ordering> ; yields void
76017610

76027611
Overview:
76037612
"""""""""
@@ -7631,17 +7640,17 @@ A ``fence`` which has ``seq_cst`` ordering, in addition to having both
76317640
``acquire`` and ``release`` semantics specified above, participates in
76327641
the global program order of other ``seq_cst`` operations and/or fences.
76337642

7634-
The optional ":ref:`singlethread <singlethread>`" argument specifies
7635-
that the fence only synchronizes with other fences in the same thread.
7636-
(This is useful for interacting with signal handlers.)
7643+
A ``fence`` instruction can also take an optional
7644+
":ref:`syncscope <syncscope>`" argument.
76377645

76387646
Example:
76397647
""""""""
76407648

76417649
.. code-block:: llvm
76427650
7643-
fence acquire ; yields void
7644-
fence singlethread seq_cst ; yields void
7651+
fence acquire ; yields void
7652+
fence syncscope("singlethread") seq_cst ; yields void
7653+
fence syncscope("agent") seq_cst ; yields void
76457654
76467655
.. _i_cmpxchg:
76477656

@@ -7653,7 +7662,7 @@ Syntax:
76537662

76547663
::
76557664

7656-
cmpxchg [weak] [volatile] <ty>* <pointer>, <ty> <cmp>, <ty> <new> [singlethread] <success ordering> <failure ordering> ; yields { ty, i1 }
7665+
cmpxchg [weak] [volatile] <ty>* <pointer>, <ty> <cmp>, <ty> <new> [syncscope("<target-scope>")] <success ordering> <failure ordering> ; yields { ty, i1 }
76577666

76587667
Overview:
76597668
"""""""""
@@ -7682,10 +7691,8 @@ must be at least ``monotonic``, the ordering constraint on failure must be no
76827691
stronger than that on success, and the failure ordering cannot be either
76837692
``release`` or ``acq_rel``.
76847693

7685-
The optional "``singlethread``" argument declares that the ``cmpxchg``
7686-
is only atomic with respect to code (usually signal handlers) running in
7687-
the same thread as the ``cmpxchg``. Otherwise the cmpxchg is atomic with
7688-
respect to all other code in the system.
7694+
A ``cmpxchg`` instruction can also take an optional
7695+
":ref:`syncscope <syncscope>`" argument.
76897696

76907697
The pointer passed into cmpxchg must have alignment greater than or
76917698
equal to the size in memory of the operand.
@@ -7739,7 +7746,7 @@ Syntax:
77397746

77407747
::
77417748

7742-
atomicrmw [volatile] <operation> <ty>* <pointer>, <ty> <value> [singlethread] <ordering> ; yields ty
7749+
atomicrmw [volatile] <operation> <ty>* <pointer>, <ty> <value> [syncscope("<target-scope>")] <ordering> ; yields ty
77437750

77447751
Overview:
77457752
"""""""""
@@ -7773,6 +7780,9 @@ be a pointer to that type. If the ``atomicrmw`` is marked as
77737780
order of execution of this ``atomicrmw`` with other :ref:`volatile
77747781
operations <volatile>`.
77757782

7783+
A ``atomicrmw`` instruction can also take an optional
7784+
":ref:`syncscope <syncscope>`" argument.
7785+
77767786
Semantics:
77777787
""""""""""
77787788

include/llvm/Bitcode/LLVMBitCodes.h

+6-6
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,8 @@ enum BlockIDs {
5959
FULL_LTO_GLOBALVAL_SUMMARY_BLOCK_ID,
6060

6161
SYMTAB_BLOCK_ID,
62+
63+
SYNC_SCOPE_NAMES_BLOCK_ID,
6264
};
6365

6466
/// Identification block contains a string that describes the producer details,
@@ -172,6 +174,10 @@ enum OperandBundleTagCode {
172174
OPERAND_BUNDLE_TAG = 1, // TAG: [strchr x N]
173175
};
174176

177+
enum SyncScopeNameCode {
178+
SYNC_SCOPE_NAME = 1,
179+
};
180+
175181
// Value symbol table codes.
176182
enum ValueSymtabCodes {
177183
VST_CODE_ENTRY = 1, // VST_ENTRY: [valueid, namechar x N]
@@ -404,12 +410,6 @@ enum AtomicOrderingCodes {
404410
ORDERING_SEQCST = 6
405411
};
406412

407-
/// Encoded SynchronizationScope values.
408-
enum AtomicSynchScopeCodes {
409-
SYNCHSCOPE_SINGLETHREAD = 0,
410-
SYNCHSCOPE_CROSSTHREAD = 1
411-
};
412-
413413
/// Markers and flags for call instruction.
414414
enum CallMarkersFlags {
415415
CALL_TAIL = 0,

include/llvm/CodeGen/MachineFunction.h

+1-1
Original file line numberDiff line numberDiff line change
@@ -650,7 +650,7 @@ class MachineFunction {
650650
MachinePointerInfo PtrInfo, MachineMemOperand::Flags f, uint64_t s,
651651
unsigned base_alignment, const AAMDNodes &AAInfo = AAMDNodes(),
652652
const MDNode *Ranges = nullptr,
653-
SynchronizationScope SynchScope = CrossThread,
653+
SyncScope::ID SSID = SyncScope::System,
654654
AtomicOrdering Ordering = AtomicOrdering::NotAtomic,
655655
AtomicOrdering FailureOrdering = AtomicOrdering::NotAtomic);
656656

include/llvm/CodeGen/MachineMemOperand.h

+6-6
Original file line numberDiff line numberDiff line change
@@ -124,8 +124,8 @@ class MachineMemOperand {
124124
private:
125125
/// Atomic information for this memory operation.
126126
struct MachineAtomicInfo {
127-
/// Synchronization scope for this memory operation.
128-
unsigned SynchScope : 1; // enum SynchronizationScope
127+
/// Synchronization scope ID for this memory operation.
128+
unsigned SSID : 8; // SyncScope::ID
129129
/// Atomic ordering requirements for this memory operation. For cmpxchg
130130
/// atomic operations, atomic ordering requirements when store occurs.
131131
unsigned Ordering : 4; // enum AtomicOrdering
@@ -152,7 +152,7 @@ class MachineMemOperand {
152152
unsigned base_alignment,
153153
const AAMDNodes &AAInfo = AAMDNodes(),
154154
const MDNode *Ranges = nullptr,
155-
SynchronizationScope SynchScope = CrossThread,
155+
SyncScope::ID SSID = SyncScope::System,
156156
AtomicOrdering Ordering = AtomicOrdering::NotAtomic,
157157
AtomicOrdering FailureOrdering = AtomicOrdering::NotAtomic);
158158

@@ -202,9 +202,9 @@ class MachineMemOperand {
202202
/// Return the range tag for the memory reference.
203203
const MDNode *getRanges() const { return Ranges; }
204204

205-
/// Return the synchronization scope for this memory operation.
206-
SynchronizationScope getSynchScope() const {
207-
return static_cast<SynchronizationScope>(AtomicInfo.SynchScope);
205+
/// Returns the synchronization scope ID for this memory operation.
206+
SyncScope::ID getSyncScopeID() const {
207+
return static_cast<SyncScope::ID>(AtomicInfo.SSID);
208208
}
209209

210210
/// Return the atomic ordering requirements for this memory operation. For

include/llvm/CodeGen/SelectionDAG.h

+2-2
Original file line numberDiff line numberDiff line change
@@ -927,7 +927,7 @@ class SelectionDAG {
927927
SDValue Cmp, SDValue Swp, MachinePointerInfo PtrInfo,
928928
unsigned Alignment, AtomicOrdering SuccessOrdering,
929929
AtomicOrdering FailureOrdering,
930-
SynchronizationScope SynchScope);
930+
SyncScope::ID SSID);
931931
SDValue getAtomicCmpSwap(unsigned Opcode, const SDLoc &dl, EVT MemVT,
932932
SDVTList VTs, SDValue Chain, SDValue Ptr,
933933
SDValue Cmp, SDValue Swp, MachineMemOperand *MMO);
@@ -937,7 +937,7 @@ class SelectionDAG {
937937
SDValue getAtomic(unsigned Opcode, const SDLoc &dl, EVT MemVT, SDValue Chain,
938938
SDValue Ptr, SDValue Val, const Value *PtrVal,
939939
unsigned Alignment, AtomicOrdering Ordering,
940-
SynchronizationScope SynchScope);
940+
SyncScope::ID SSID);
941941
SDValue getAtomic(unsigned Opcode, const SDLoc &dl, EVT MemVT, SDValue Chain,
942942
SDValue Ptr, SDValue Val, MachineMemOperand *MMO);
943943

include/llvm/CodeGen/SelectionDAGNodes.h

+2-2
Original file line numberDiff line numberDiff line change
@@ -1213,8 +1213,8 @@ class MemSDNode : public SDNode {
12131213
/// Returns the Ranges that describes the dereference.
12141214
const MDNode *getRanges() const { return MMO->getRanges(); }
12151215

1216-
/// Return the synchronization scope for this memory operation.
1217-
SynchronizationScope getSynchScope() const { return MMO->getSynchScope(); }
1216+
/// Returns the synchronization scope ID for this memory operation.
1217+
SyncScope::ID getSyncScopeID() const { return MMO->getSyncScopeID(); }
12181218

12191219
/// Return the atomic ordering requirements for this memory operation. For
12201220
/// cmpxchg atomic operations, return the atomic ordering requirements when

include/llvm/IR/IRBuilder.h

+6-6
Original file line numberDiff line numberDiff line change
@@ -1203,22 +1203,22 @@ class IRBuilder : public IRBuilderBase, public Inserter {
12031203
return SI;
12041204
}
12051205
FenceInst *CreateFence(AtomicOrdering Ordering,
1206-
SynchronizationScope SynchScope = CrossThread,
1206+
SyncScope::ID SSID = SyncScope::System,
12071207
const Twine &Name = "") {
1208-
return Insert(new FenceInst(Context, Ordering, SynchScope), Name);
1208+
return Insert(new FenceInst(Context, Ordering, SSID), Name);
12091209
}
12101210
AtomicCmpXchgInst *
12111211
CreateAtomicCmpXchg(Value *Ptr, Value *Cmp, Value *New,
12121212
AtomicOrdering SuccessOrdering,
12131213
AtomicOrdering FailureOrdering,
1214-
SynchronizationScope SynchScope = CrossThread) {
1214+
SyncScope::ID SSID = SyncScope::System) {
12151215
return Insert(new AtomicCmpXchgInst(Ptr, Cmp, New, SuccessOrdering,
1216-
FailureOrdering, SynchScope));
1216+
FailureOrdering, SSID));
12171217
}
12181218
AtomicRMWInst *CreateAtomicRMW(AtomicRMWInst::BinOp Op, Value *Ptr, Value *Val,
12191219
AtomicOrdering Ordering,
1220-
SynchronizationScope SynchScope = CrossThread) {
1221-
return Insert(new AtomicRMWInst(Op, Ptr, Val, Ordering, SynchScope));
1220+
SyncScope::ID SSID = SyncScope::System) {
1221+
return Insert(new AtomicRMWInst(Op, Ptr, Val, Ordering, SSID));
12221222
}
12231223
Value *CreateGEP(Value *Ptr, ArrayRef<Value *> IdxList,
12241224
const Twine &Name = "") {

0 commit comments

Comments
 (0)